Oct 31 22:47:08.419768 ip-10-0-135-163 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 31 22:47:08.419782 ip-10-0-135-163 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 31 22:47:08.419792 ip-10-0-135-163 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 31 22:47:08.420125 ip-10-0-135-163 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 31 22:47:18.668809 ip-10-0-135-163 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 31 22:47:18.668824 ip-10-0-135-163 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 790950323b68469b9072952d60400029 --
Oct 31 22:49:01.110649 ip-10-0-135-163 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 22:49:01.567491 ip-10-0-135-163 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 22:49:01.567491 ip-10-0-135-163 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 22:49:01.567491 ip-10-0-135-163 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 22:49:01.567491 ip-10-0-135-163 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 22:49:01.570754 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570674    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 22:49:01.570754 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570754    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 22:49:01.570821 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570760    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 22:49:01.570821 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570765    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 22:49:01.570821 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570769    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 22:49:01.570821 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570772    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 22:49:01.570821 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570775    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 22:49:01.570821 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570778    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 22:49:01.570821 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570780    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 22:49:01.570821 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570783    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 22:49:01.570821 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570786    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 22:49:01.570821 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570789    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 22:49:01.570821 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570791    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 22:49:01.570821 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570794    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 22:49:01.570821 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570796    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 22:49:01.570821 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570799    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 22:49:01.570821 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570801    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 22:49:01.570821 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570804    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 22:49:01.570821 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570806    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 22:49:01.570821 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570808    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 22:49:01.570821 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570811    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 22:49:01.571272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570813    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 22:49:01.571272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570816    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 22:49:01.571272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570819    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 22:49:01.571272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570821    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 22:49:01.571272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570824    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 22:49:01.571272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570827    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 22:49:01.571272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570829    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 22:49:01.571272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570838    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 22:49:01.571272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570840    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 22:49:01.571272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570843    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 22:49:01.571272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570846    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 22:49:01.571272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570850    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 22:49:01.571272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570853    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 22:49:01.571272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570855    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 22:49:01.571272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570858    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 22:49:01.571272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570861    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 22:49:01.571272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570863    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 22:49:01.571272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570866    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 22:49:01.571272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570868    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 22:49:01.571736 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570871    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 22:49:01.571736 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570873    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 22:49:01.571736 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570876    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 22:49:01.571736 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570878    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 22:49:01.571736 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570882    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 22:49:01.571736 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570884    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 22:49:01.571736 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570887    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 22:49:01.571736 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570889    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 22:49:01.571736 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570893    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 22:49:01.571736 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570896    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 22:49:01.571736 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570900    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 22:49:01.571736 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570903    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 22:49:01.571736 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570905    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 22:49:01.571736 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570908    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 22:49:01.571736 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570911    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 22:49:01.571736 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570914    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 22:49:01.571736 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570916    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 22:49:01.571736 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570919    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 22:49:01.571736 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570922    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 22:49:01.572184 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570924    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 22:49:01.572184 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570927    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 22:49:01.572184 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570929    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 22:49:01.572184 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570932    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 22:49:01.572184 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570934    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 22:49:01.572184 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570936    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 22:49:01.572184 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570939    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 22:49:01.572184 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570941    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 22:49:01.572184 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570944    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 22:49:01.572184 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570946    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 22:49:01.572184 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570948    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 22:49:01.572184 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570950    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 22:49:01.572184 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570953    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 22:49:01.572184 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570955    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 22:49:01.572184 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570958    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 22:49:01.572184 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570960    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 22:49:01.572184 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570962    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 22:49:01.572184 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570965    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 22:49:01.572184 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570967    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 22:49:01.572184 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570970    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 22:49:01.572755 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570972    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 22:49:01.572755 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570975    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 22:49:01.572755 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570977    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 22:49:01.572755 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570980    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 22:49:01.572755 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570983    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 22:49:01.572755 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570985    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 22:49:01.572755 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570988    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 22:49:01.572755 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570991    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 22:49:01.572755 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570994    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 22:49:01.572755 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570996    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 22:49:01.572755 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.570999    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 22:49:01.572755 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.571001    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 22:49:01.572755 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.571004    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 22:49:01.572755 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.571006    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 22:49:01.572755 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.571009    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 22:49:01.572755 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.571011    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 22:49:01.572755 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.571013    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 22:49:01.572755 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.571016    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 22:49:01.572755 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.571020    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 22:49:01.573227 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.571022    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 22:49:01.573227 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.571025    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 22:49:01.573227 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.571027    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 22:49:01.573227 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.571030    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 22:49:01.573227 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.571032    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 22:49:01.573227 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.571034    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 22:49:01.573227 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.571037    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 22:49:01.573227 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.571039    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 22:49:01.573444 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573438    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 22:49:01.573472 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573447    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 22:49:01.573472 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573451    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 22:49:01.573472 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573454    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 22:49:01.573472 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573457    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 22:49:01.573472 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573460    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 22:49:01.573472 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573462    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 22:49:01.573472 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573465    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 22:49:01.573472 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573468    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 22:49:01.573472 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573471    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 22:49:01.573472 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573474    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 22:49:01.573472 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573477    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 22:49:01.573740 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573480    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 22:49:01.573740 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573483    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 22:49:01.573740 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573485    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 22:49:01.573740 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573488    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 22:49:01.573740 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573491    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 22:49:01.573740 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573493    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 22:49:01.573740 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573496    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 22:49:01.573740 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573500    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 22:49:01.573740 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573503    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 22:49:01.573740 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573506    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 22:49:01.573740 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573508    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 22:49:01.573740 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573512    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 22:49:01.573740 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573516    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 22:49:01.573740 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573519    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 22:49:01.573740 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573522    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 22:49:01.573740 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573524    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 22:49:01.573740 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573527    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 22:49:01.573740 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573530    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 22:49:01.574154 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573532    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 22:49:01.574154 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573534    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 22:49:01.574154 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573537    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 22:49:01.574154 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573539    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 22:49:01.574154 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573542    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 22:49:01.574154 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573544    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 22:49:01.574154 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573547    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 22:49:01.574154 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573549    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 22:49:01.574154 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573552    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 22:49:01.574154 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573555    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 22:49:01.574154 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573557    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 22:49:01.574154 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573567    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 22:49:01.574154 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573570    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 22:49:01.574154 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573573    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 22:49:01.574154 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573575    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 22:49:01.574154 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573579    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 22:49:01.574154 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573582    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 22:49:01.574154 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573584    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 22:49:01.574154 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573587    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 22:49:01.574154 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573589    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 22:49:01.574624 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573592    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 22:49:01.574624 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573594    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 22:49:01.574624 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573597    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 22:49:01.574624 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573599    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 22:49:01.574624 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573602    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 22:49:01.574624 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573604    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 22:49:01.574624 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573607    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 22:49:01.574624 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573609    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 22:49:01.574624 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573612    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 22:49:01.574624 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573614    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 22:49:01.574624 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573617    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 22:49:01.574624 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573619    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 22:49:01.574624 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573622    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 22:49:01.574624 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573624    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 22:49:01.574624 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573626    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 22:49:01.574624 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573629    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 22:49:01.574624 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573631    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 22:49:01.574624 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573633    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 22:49:01.574624 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573636    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 22:49:01.574624 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573639    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 22:49:01.575085 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573641    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 22:49:01.575085 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573644    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 22:49:01.575085 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573646    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 22:49:01.575085 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573648    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 22:49:01.575085 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573656    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 22:49:01.575085 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573659    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 22:49:01.575085 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573662    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 22:49:01.575085 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573665    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 22:49:01.575085 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573668    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 22:49:01.575085 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573670    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 22:49:01.575085 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573673    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 22:49:01.575085 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573675    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 22:49:01.575085 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573677    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 22:49:01.575085 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573680    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 22:49:01.575085 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573682    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 22:49:01.575085 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573685    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 22:49:01.575085 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573687    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 22:49:01.575085 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573689    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 22:49:01.575085 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573692    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 22:49:01.575574 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573694    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 22:49:01.575574 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573696    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 22:49:01.575574 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573699    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 22:49:01.575574 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573701    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 22:49:01.575574 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573704    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 22:49:01.575574 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573706    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 22:49:01.575574 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573708    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 22:49:01.575574 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573711    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 22:49:01.575574 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573713    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 22:49:01.575574 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573715    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 22:49:01.575574 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573718    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 22:49:01.575574 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573721    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 22:49:01.575574 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573723    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 22:49:01.575574 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573726    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 22:49:01.575574 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573729    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 22:49:01.575574 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573732    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 22:49:01.575574 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.573734    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 22:49:01.575574 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574260    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 22:49:01.575574 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574266    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 22:49:01.575574 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574270    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 22:49:01.576056 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574274    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 22:49:01.576056 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574276    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 22:49:01.576056 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574279    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 22:49:01.576056 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574283    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 22:49:01.576056 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574287    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 22:49:01.576056 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574289    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 22:49:01.576056 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574293    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 22:49:01.576056 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574295    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 22:49:01.576056 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574298    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 22:49:01.576056 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574300    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 22:49:01.576056 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574303    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 22:49:01.576056 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574305    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 22:49:01.576056 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574308    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 22:49:01.576056 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574312    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 22:49:01.576056 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574316    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 22:49:01.576056 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574319    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 22:49:01.576056 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574322    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 22:49:01.576056 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574325    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 22:49:01.576056 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574328    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 22:49:01.576530 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574331    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 22:49:01.576530 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574333    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 22:49:01.576530 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574336    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 22:49:01.576530 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574338    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 22:49:01.576530 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574341    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 22:49:01.576530 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574344    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 22:49:01.576530 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574346    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 22:49:01.576530 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574348    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 22:49:01.576530 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574351    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 22:49:01.576530 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574366    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 22:49:01.576530 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574369    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 22:49:01.576530 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574373    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 22:49:01.576530 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574375    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 22:49:01.576530 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574378    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 22:49:01.576530 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574381    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 22:49:01.576530 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574384    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 22:49:01.576530 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574386    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 22:49:01.576530 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574389    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 22:49:01.576530 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574391    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 22:49:01.576530 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574394    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 22:49:01.576997 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574396    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 22:49:01.576997 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574399    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 22:49:01.576997 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574401    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 22:49:01.576997 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574404    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 22:49:01.576997 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574406    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 22:49:01.576997 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574409    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 22:49:01.576997 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574411    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 22:49:01.576997 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574414    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 22:49:01.576997 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574416    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 22:49:01.576997 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574419    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 22:49:01.576997 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574421    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 22:49:01.576997 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574424    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 22:49:01.576997 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574426    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 22:49:01.576997 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574428    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 22:49:01.576997 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574431    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 22:49:01.576997 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574433    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 22:49:01.576997 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574436    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 22:49:01.576997 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574439    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 22:49:01.576997 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574441    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 22:49:01.576997 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574444    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 22:49:01.577482 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574446    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 22:49:01.577482 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574448    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 22:49:01.577482 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574451    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 22:49:01.577482 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574453    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 22:49:01.577482 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574456    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 22:49:01.577482 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574459    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 22:49:01.577482 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574462    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 22:49:01.577482 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574464    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 22:49:01.577482 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574467    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 22:49:01.577482 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574469    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 22:49:01.577482 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574471    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 22:49:01.577482 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574474    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 22:49:01.577482 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574476    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 22:49:01.577482 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574479    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 22:49:01.577482 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574481    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 22:49:01.577482 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574483    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 22:49:01.577482 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574486    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 22:49:01.577482 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574488    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 22:49:01.577482 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574490    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 22:49:01.577935 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574493    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 22:49:01.577935 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574496    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 22:49:01.577935 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574498    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 22:49:01.577935 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574500    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 22:49:01.577935 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574503    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 22:49:01.577935 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574506    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 22:49:01.577935 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574508    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 22:49:01.577935 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574510    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 22:49:01.577935 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574513    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 22:49:01.577935 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574516    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 22:49:01.577935 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574519    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 22:49:01.577935 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574521    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 22:49:01.577935 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574524    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 22:49:01.577935 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574526    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 22:49:01.577935 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574528    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 22:49:01.577935 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574531    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 22:49:01.577935 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574533    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 22:49:01.577935 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574536    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 22:49:01.577935 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574539    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 22:49:01.578387 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574542    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 22:49:01.578387 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574544    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 22:49:01.578387 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574547    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 22:49:01.578387 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574549    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 22:49:01.578387 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574552    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 22:49:01.578387 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.574554    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 22:49:01.578387 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575592    2575 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 22:49:01.578387 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575603    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 22:49:01.578387 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575609    2575 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 22:49:01.578387 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575614    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 22:49:01.578387 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575619    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 22:49:01.578387 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575623    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 22:49:01.578387 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575627    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 22:49:01.578387 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575632    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 22:49:01.578387 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575635    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 22:49:01.578387 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575638    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 22:49:01.578387 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575641    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 22:49:01.578387 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575645    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 22:49:01.578387 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575648    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 22:49:01.578387 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575651    2575 flags.go:64] FLAG: --cgroup-root=""
Oct 31 22:49:01.578387 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575654    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 22:49:01.578387 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575657    2575 flags.go:64] FLAG: --client-ca-file=""
Oct 31 22:49:01.578894 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575661    2575 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 22:49:01.578894 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575664    2575 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 22:49:01.578894 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575669    2575 flags.go:64] FLAG: --cluster-domain=""
Oct 31 22:49:01.578894 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575672    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 22:49:01.578894 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575675    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 22:49:01.578894 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575679    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 22:49:01.578894 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575683    2575 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 22:49:01.578894 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575687    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 22:49:01.578894 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575690    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 22:49:01.578894 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575693    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 22:49:01.578894 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575697    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 22:49:01.578894 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575700    2575 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 22:49:01.578894 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575703    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 22:49:01.578894 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575707    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 22:49:01.578894 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575710    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 22:49:01.578894 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575713    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 22:49:01.578894 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575717    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 22:49:01.578894 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575720    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 22:49:01.578894 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575723    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 22:49:01.578894 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575726    2575 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 22:49:01.578894 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575729    2575 flags.go:64] FLAG: --enable-server="true"
Oct 31 22:49:01.578894 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575732    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 22:49:01.578894 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575737    2575 flags.go:64] FLAG: --event-burst="100"
Oct 31 22:49:01.578894 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575740    2575 flags.go:64] FLAG: --event-qps="50"
Oct 31 22:49:01.578894 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575743    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 22:49:01.579529 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575746    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 22:49:01.579529 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575748    2575 flags.go:64] FLAG: --eviction-hard=""
Oct 31 22:49:01.579529 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575753    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 22:49:01.579529 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575756    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 22:49:01.579529 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575759    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 22:49:01.579529 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575762    2575 flags.go:64] FLAG: --eviction-soft=""
Oct 31 22:49:01.579529 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575765    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 22:49:01.579529 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575769    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 22:49:01.579529 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575772    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 22:49:01.579529 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575775    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 22:49:01.579529 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575778    2575 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 22:49:01.579529 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575781    2575 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 22:49:01.579529 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575784    2575 flags.go:64] FLAG: --feature-gates=""
Oct 31 22:49:01.579529 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575791    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 22:49:01.579529 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575794    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 22:49:01.579529 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575798    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 22:49:01.579529 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575801    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 22:49:01.579529 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575804    2575 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 22:49:01.579529 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575808    2575 flags.go:64] FLAG: --help="false"
Oct 31 22:49:01.579529 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575811    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:01.579529 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575815    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 22:49:01.579529 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575818    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 22:49:01.579529 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575821    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 22:49:01.580109 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575824    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 22:49:01.580109 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575827    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 22:49:01.580109 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575830    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 22:49:01.580109 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575833    2575 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 22:49:01.580109 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575836    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 22:49:01.580109 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575839    2575 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 22:49:01.580109 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575842    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 22:49:01.580109 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575845    2575 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 22:49:01.580109 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575847    2575 flags.go:64] FLAG: --kube-reserved=""
Oct 31 22:49:01.580109 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575850    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 22:49:01.580109 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575853    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 22:49:01.580109 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575856    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 22:49:01.580109 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575859    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 22:49:01.580109 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575862    2575 flags.go:64] FLAG: --lock-file=""
Oct 31 22:49:01.580109 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575865    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 22:49:01.580109 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575868    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 22:49:01.580109 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575871    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 22:49:01.580109 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575877    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 22:49:01.580109 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575880    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 22:49:01.580109 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575884    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 22:49:01.580109 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575887    2575 flags.go:64] FLAG: --logging-format="text"
Oct 31 22:49:01.580109 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575890    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 22:49:01.580109 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575893    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 22:49:01.580109 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575896    2575 flags.go:64] FLAG: --manifest-url=""
Oct 31 22:49:01.580708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575899    2575 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 22:49:01.580708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575904    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 22:49:01.580708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575907    2575 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 22:49:01.580708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575910    2575 flags.go:64] FLAG: --max-pods="110"
Oct 31 22:49:01.580708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575914    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 22:49:01.580708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575917    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 22:49:01.580708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575920    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 22:49:01.580708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575923    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 22:49:01.580708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575925    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 22:49:01.580708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575928    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 22:49:01.580708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575931    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 22:49:01.580708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575938    2575 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 22:49:01.580708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575941    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 22:49:01.580708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575944    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 22:49:01.580708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575947    2575 flags.go:64] FLAG: --pod-cidr=""
Oct 31 22:49:01.580708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575950    2575 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 22:49:01.580708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575953    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 22:49:01.580708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575956    2575 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 22:49:01.580708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575959    2575 flags.go:64] FLAG: --port="10250"
Oct 31 22:49:01.580708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575962    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 22:49:01.580708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575965    2575 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0d2c56eb0f44aa2ca"
Oct 31 22:49:01.580708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575969    2575 flags.go:64] FLAG: --qos-reserved=""
Oct 31 22:49:01.580708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575972    2575 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 22:49:01.580708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575975    2575 flags.go:64] FLAG: --register-node="true"
Oct 31 22:49:01.580708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575978    2575 flags.go:64] FLAG: --register-with-taints=""
Oct 31 22:49:01.580708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575981    2575 flags.go:64] FLAG: --registry-burst="10"
Oct 31 22:49:01.581339 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575984    2575 flags.go:64] FLAG: --registry-qps="5"
Oct 31 22:49:01.581339 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575987    2575 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 22:49:01.581339 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575990    2575 flags.go:64] FLAG: --reserved-memory=""
Oct 31 22:49:01.581339 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575994    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 22:49:01.581339 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.575997    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 22:49:01.581339 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.576000    2575 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 22:49:01.581339 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.576003    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 22:49:01.581339 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.576008    2575 flags.go:64] FLAG: --runonce="false"
Oct 31 22:49:01.581339 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.576012    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 22:49:01.581339 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.576015    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 22:49:01.581339 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.576019    2575 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 22:49:01.581339 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.576022    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 22:49:01.581339 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.576025    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 22:49:01.581339 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.576029    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 22:49:01.581339 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.576032    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 22:49:01.581339 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.576035    2575 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 22:49:01.581339 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.576038    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 22:49:01.581339 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.576041    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 22:49:01.581339 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.576044    2575 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 22:49:01.581339 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.576047    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 22:49:01.581339 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.576050    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 22:49:01.581339 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.576053    2575 flags.go:64] FLAG: --system-cgroups=""
Oct 31 22:49:01.581339 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.576055    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 22:49:01.581339 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.576061    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 22:49:01.581339 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.576064    2575 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 22:49:01.581926 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.576067    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 22:49:01.581926 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.576071    2575 flags.go:64] FLAG: --tls-min-version=""
Oct 31 22:49:01.581926 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.576075    2575 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 22:49:01.581926 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.576078    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 22:49:01.581926 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.576081    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 22:49:01.581926 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.576086    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 22:49:01.581926 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.576089    2575 flags.go:64] FLAG: --v="2"
Oct 31 22:49:01.581926 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.576094    2575 flags.go:64] FLAG: --version="false"
Oct 31 22:49:01.581926 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.576098    2575 flags.go:64] FLAG: --vmodule=""
Oct 31 22:49:01.581926 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.576102    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 22:49:01.581926 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.576105    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 22:49:01.581926 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576251    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 22:49:01.581926 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576256    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 22:49:01.581926 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576258    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 22:49:01.581926 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576262    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 22:49:01.581926 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576266    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 22:49:01.581926 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576268    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 22:49:01.581926 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576271    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 22:49:01.581926 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576274    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 22:49:01.581926 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576277    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 22:49:01.581926 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576280    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 22:49:01.581926 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576282    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 22:49:01.582490 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576285    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 22:49:01.582490 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576288    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 22:49:01.582490 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576290    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 22:49:01.582490 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576293    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 22:49:01.582490 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576295    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 22:49:01.582490 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576298    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 22:49:01.582490 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576300    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 22:49:01.582490 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576303    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 22:49:01.582490 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576306    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 22:49:01.582490 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576308    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 22:49:01.582490 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576311    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 22:49:01.582490 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576313    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 22:49:01.582490 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576316    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 22:49:01.582490 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576318    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 22:49:01.582490 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576321    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 22:49:01.582490 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576325    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 22:49:01.582490 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576328    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 22:49:01.582490 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576331    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 22:49:01.582490 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576333    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 22:49:01.582490 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576336    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 22:49:01.582963 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576339    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 22:49:01.582963 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576341    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 22:49:01.582963 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576344    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 22:49:01.582963 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576346    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 22:49:01.582963 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576350    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 22:49:01.582963 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576353    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 22:49:01.582963 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576367    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 22:49:01.582963 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576370    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 22:49:01.582963 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576373    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 22:49:01.582963 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576376    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 22:49:01.582963 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576379    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 22:49:01.582963 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576382    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 22:49:01.582963 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576385    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 22:49:01.582963 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576388    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 22:49:01.582963 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576391    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 22:49:01.582963 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576393    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 22:49:01.582963 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576396    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 22:49:01.582963 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576399    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 22:49:01.582963 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576401    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 22:49:01.583445 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576404    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 22:49:01.583445 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576406    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 22:49:01.583445 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576409    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 22:49:01.583445 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576413    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 22:49:01.583445 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576417    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 22:49:01.583445 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576420    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 22:49:01.583445 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576434    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 22:49:01.583445 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576437    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 22:49:01.583445 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576442    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 22:49:01.583445 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576446    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 22:49:01.583445 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576448    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 22:49:01.583445 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576451    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 22:49:01.583445 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576454    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 22:49:01.583445 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576457    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 22:49:01.583445 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576459    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 22:49:01.583445 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576461    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 22:49:01.583445 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576464    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 22:49:01.583445 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576468    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 22:49:01.583445 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576470    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 22:49:01.583894 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576473    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 22:49:01.583894 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576475    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 22:49:01.583894 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576478    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 22:49:01.583894 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576480    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 22:49:01.583894 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576483    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 22:49:01.583894 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576486    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 22:49:01.583894 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576488    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 22:49:01.583894 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576491    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 22:49:01.583894 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576493    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 22:49:01.583894 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576496    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 22:49:01.583894 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576499    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 22:49:01.583894 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576502    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 22:49:01.583894 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576505    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 22:49:01.583894 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576508    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 22:49:01.583894 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576510    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 22:49:01.583894 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576513    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 22:49:01.583894 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576515    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 22:49:01.583894 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576518    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 22:49:01.583894 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576521    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 22:49:01.583894 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576523    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 22:49:01.584508 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576526    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 22:49:01.584508 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576530    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 22:49:01.584508 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576533    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 22:49:01.584508 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576535    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 22:49:01.584508 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576538    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 22:49:01.584508 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576541    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 22:49:01.584508 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576544    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 22:49:01.584508 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576546    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 22:49:01.584508 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576549    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 22:49:01.584508 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576551    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 22:49:01.584508 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576555    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 22:49:01.584508 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576558    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 22:49:01.584508 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576560    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 22:49:01.584508 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576563    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 22:49:01.584508 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576565    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 22:49:01.584508 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576569    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 22:49:01.584508 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576572    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 22:49:01.584902 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.576598    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 22:49:01.584902 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576638    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 22:49:01.584902 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576642    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 22:49:01.584902 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576663    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 22:49:01.584902 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576683    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 22:49:01.584902 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576688    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 22:49:01.584902 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576691    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 22:49:01.584902 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576694    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 22:49:01.584902 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576697    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 22:49:01.584902 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576702    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 22:49:01.584902 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576705    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 22:49:01.584902 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576708    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 22:49:01.584902 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576710    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 22:49:01.584902 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576714    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 22:49:01.584902 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576717    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 22:49:01.584902 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576721    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 22:49:01.585272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576726    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 22:49:01.585272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576729    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 22:49:01.585272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576732    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 22:49:01.585272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576735    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 22:49:01.585272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576737    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 22:49:01.585272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576740    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 22:49:01.585272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576743    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 22:49:01.585272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576746    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 22:49:01.585272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576749    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 22:49:01.585272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576752    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 22:49:01.585272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576755    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 22:49:01.585272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576758    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 22:49:01.585272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576760    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 22:49:01.585272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576763    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 22:49:01.585272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576766    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 22:49:01.585272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576768    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 22:49:01.585272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576771    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 22:49:01.585272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576774    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 22:49:01.585272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576776    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 22:49:01.585272 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576782    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 22:49:01.585744 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576785    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 22:49:01.585744 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576788    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 22:49:01.585744 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576791    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 22:49:01.585744 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576794    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 22:49:01.585744 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576796    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 22:49:01.585744 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576799    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 22:49:01.585744 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576802    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 22:49:01.585744 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576804    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 22:49:01.585744 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576807    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 22:49:01.585744 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576810    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 22:49:01.585744 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576813    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 22:49:01.585744 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576815    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 22:49:01.585744 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576818    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 22:49:01.585744 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576820    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 22:49:01.585744 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576822    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 22:49:01.585744 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576825    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 22:49:01.585744 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576827    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 22:49:01.585744 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576830    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 22:49:01.585744 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576832    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 22:49:01.585744 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576835    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 22:49:01.586194 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576837    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 22:49:01.586194 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576840    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 22:49:01.586194 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576842    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 22:49:01.586194 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576845    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 22:49:01.586194 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576847    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 22:49:01.586194 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576850    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 22:49:01.586194 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576852    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 22:49:01.586194 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576854    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 22:49:01.586194 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576857    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 22:49:01.586194 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576860    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 22:49:01.586194 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576862    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 22:49:01.586194 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576866    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 22:49:01.586194 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576873    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 22:49:01.586194 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576876    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 22:49:01.586194 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576880    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 22:49:01.586194 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576882    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 22:49:01.586194 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576885    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 22:49:01.586194 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576887    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 22:49:01.586194 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576890    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 22:49:01.586194 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576892    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 22:49:01.586685 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576896    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 22:49:01.586685 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576898    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 22:49:01.586685 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576900    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 22:49:01.586685 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576903    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 22:49:01.586685 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576905    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 22:49:01.586685 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576908    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 22:49:01.586685 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576911    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 22:49:01.586685 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576913    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 22:49:01.586685 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576916    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 22:49:01.586685 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576918    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 22:49:01.586685 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576921    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 22:49:01.586685 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576924    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 22:49:01.586685 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576926    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 22:49:01.586685 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576929    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 22:49:01.586685 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576931    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 22:49:01.586685 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576934    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 22:49:01.586685 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576936    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 22:49:01.586685 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576938    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 22:49:01.586685 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576941    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 22:49:01.587114 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576951    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 22:49:01.587114 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576956    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 22:49:01.587114 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576959    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 22:49:01.587114 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576962    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 22:49:01.587114 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576966    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 22:49:01.587114 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576970    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 22:49:01.587114 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576973    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 22:49:01.587114 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576975    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 22:49:01.587114 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576978    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 22:49:01.587114 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576981    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 22:49:01.587114 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576983    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 22:49:01.587114 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.576986    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 22:49:01.587114 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.584551    2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 22:49:01.587114 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.584569    2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 22:49:01.587114 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.584585    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 22:49:01.587114 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.584589    2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 22:49:01.587114 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584674    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 22:49:01.587114 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584679    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 22:49:01.587114 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584682    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 22:49:01.587599 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584685    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 22:49:01.587599 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584688    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 22:49:01.587599 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584691    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 22:49:01.587599 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584693    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 22:49:01.587599 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584696    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 22:49:01.587599 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584699    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 22:49:01.587599 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584701    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 22:49:01.587599 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584704    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 22:49:01.587599 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584706    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 22:49:01.587599 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584709    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 22:49:01.587599 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584711    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 22:49:01.587599 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584714    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 22:49:01.587599 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584716    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 22:49:01.587599 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584719    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 22:49:01.587599 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584721    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 22:49:01.587599 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584724    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 22:49:01.587599 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584726    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 22:49:01.587599 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584729    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 22:49:01.587599 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584731    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 22:49:01.588044 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584734    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 22:49:01.588044 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584736    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 22:49:01.588044 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584745    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 22:49:01.588044 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584747    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 22:49:01.588044 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584750    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 22:49:01.588044 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584753    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 22:49:01.588044 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584755    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 22:49:01.588044 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584758    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 22:49:01.588044 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584761    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 22:49:01.588044 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584764    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 22:49:01.588044 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584766    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 22:49:01.588044 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584769    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 22:49:01.588044 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584771    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 22:49:01.588044 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584773    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 22:49:01.588044 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584776    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 22:49:01.588044 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584778    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 22:49:01.588044 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584781    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 22:49:01.588044 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584784    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 22:49:01.588044 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584786    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 22:49:01.588044 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584789    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 22:49:01.588585 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584792    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 22:49:01.588585 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584794    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 22:49:01.588585 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584796    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 22:49:01.588585 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584799    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 22:49:01.588585 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584802    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 22:49:01.588585 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584804    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 22:49:01.588585 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584808    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 22:49:01.588585 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584813    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 22:49:01.588585 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584815    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 22:49:01.588585 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584818    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 22:49:01.588585 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584822    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 22:49:01.588585 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584825    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 22:49:01.588585 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584828    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 22:49:01.588585 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584831    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 22:49:01.588585 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584834    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 22:49:01.588585 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584843    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 22:49:01.588585 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584845    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 22:49:01.588585 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584848    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 22:49:01.588585 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584851    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 22:49:01.589023 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584854    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 22:49:01.589023 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584856    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 22:49:01.589023 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584859    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 22:49:01.589023 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584862    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 22:49:01.589023 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584864    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 22:49:01.589023 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584867    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 22:49:01.589023 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584870    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 22:49:01.589023 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584872    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 22:49:01.589023 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584875    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 22:49:01.589023 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584877    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 22:49:01.589023 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584880    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 22:49:01.589023 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584882    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 22:49:01.589023 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584884    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 22:49:01.589023 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584887    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 22:49:01.589023 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584890    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 22:49:01.589023 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584892    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 22:49:01.589023 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584895    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 22:49:01.589023 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584897    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 22:49:01.589023 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584900    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 22:49:01.589023 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584902    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 22:49:01.589590 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584905    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 22:49:01.589590 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584907    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 22:49:01.589590 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584910    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 22:49:01.589590 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584919    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 22:49:01.589590 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584922    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 22:49:01.589590 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584924    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 22:49:01.589590 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584927    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 22:49:01.589590 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584929    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 22:49:01.589590 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584932    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 22:49:01.589590 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584939    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 22:49:01.589590 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584942    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 22:49:01.589590 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584945    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 22:49:01.589590 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584948    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 22:49:01.589590 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584950    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 22:49:01.589590 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584952    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 22:49:01.589590 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584955    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 22:49:01.589590 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584957    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 22:49:01.589590 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584960    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 22:49:01.589590 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584963    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 22:49:01.590031 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584966    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 22:49:01.590031 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584968    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 22:49:01.590031 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584970    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 22:49:01.590031 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584973    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 22:49:01.590031 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584975    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 22:49:01.590031 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.584978    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 22:49:01.590031 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.585006    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 22:49:01.590031 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585189    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 22:49:01.590031 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585194    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 22:49:01.590031 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585196    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 22:49:01.590031 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585199    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 22:49:01.590031 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585201    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 22:49:01.590031 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585204    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 22:49:01.590031 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585206    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 22:49:01.590031 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585209    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 22:49:01.590031 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585211    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 22:49:01.590404 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585214    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 22:49:01.590404 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585216    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 22:49:01.590404 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585219    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 22:49:01.590404 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585221    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 22:49:01.590404 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585223    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 22:49:01.590404 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585226    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 22:49:01.590404 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585228    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 22:49:01.590404 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585236    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 22:49:01.590404 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585239    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 22:49:01.590404 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585242    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 22:49:01.590404 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585244    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 22:49:01.590404 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585246    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 22:49:01.590404 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585249    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 22:49:01.590404 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585251    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 22:49:01.590404 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585254    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 22:49:01.590404 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585256    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 22:49:01.590404 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585259    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 22:49:01.590404 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585261    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 22:49:01.590404 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585264    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 22:49:01.590404 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585267    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 22:49:01.590909 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585269    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 22:49:01.590909 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585272    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 22:49:01.590909 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585274    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 22:49:01.590909 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585277    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 22:49:01.590909 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585280    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 22:49:01.590909 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585282    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 22:49:01.590909 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585285    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 22:49:01.590909 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585287    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 22:49:01.590909 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585290    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 22:49:01.590909 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585294    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 22:49:01.590909 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585298    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 22:49:01.590909 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585301    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 22:49:01.590909 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585304    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 22:49:01.590909 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585306    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 22:49:01.590909 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585309    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 22:49:01.590909 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585311    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 22:49:01.590909 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585314    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 22:49:01.590909 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585316    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 22:49:01.590909 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585318    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 22:49:01.590909 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585321    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 22:49:01.591468 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585329    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 22:49:01.591468 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585332    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 22:49:01.591468 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585334    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 22:49:01.591468 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585337    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 22:49:01.591468 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585340    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 22:49:01.591468 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585342    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 22:49:01.591468 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585344    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 22:49:01.591468 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585347    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 22:49:01.591468 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585349    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 22:49:01.591468 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585352    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 22:49:01.591468 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585367    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 22:49:01.591468 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585370    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 22:49:01.591468 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585372    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 22:49:01.591468 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585375    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 22:49:01.591468 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585377    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 22:49:01.591468 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585379    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 22:49:01.591468 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585382    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 22:49:01.591468 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585384    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 22:49:01.591468 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585387    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 22:49:01.592093 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585390    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 22:49:01.592093 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585392    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 22:49:01.592093 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585395    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 22:49:01.592093 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585397    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 22:49:01.592093 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585400    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 22:49:01.592093 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585402    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 22:49:01.592093 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585405    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 22:49:01.592093 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585407    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 22:49:01.592093 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585410    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 22:49:01.592093 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585412    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 22:49:01.592093 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585414    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 22:49:01.592093 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585417    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 22:49:01.592093 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585419    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 22:49:01.592093 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585422    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 22:49:01.592093 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585431    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 22:49:01.592093 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585435    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 22:49:01.592093 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585437    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 22:49:01.592093 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585440    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 22:49:01.592093 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585442    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 22:49:01.592615 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585444    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 22:49:01.592615 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585448    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 22:49:01.592615 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585450    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 22:49:01.592615 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585453    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 22:49:01.592615 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585455    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 22:49:01.592615 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585457    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 22:49:01.592615 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585460    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 22:49:01.592615 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585462    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 22:49:01.592615 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585465    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 22:49:01.592615 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585467    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 22:49:01.592615 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585469    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 22:49:01.592615 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585472    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 22:49:01.592615 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585474    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 22:49:01.592615 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585477    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 22:49:01.592615 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585479    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 22:49:01.592615 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585482    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 22:49:01.592615 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585484    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 22:49:01.592615 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585487    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 22:49:01.592615 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585489    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 22:49:01.593106 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.585512    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 22:49:01.593106 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585549    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 22:49:01.593106 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585554    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 22:49:01.593106 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585557    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 22:49:01.593106 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585559    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 22:49:01.593106 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585562    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 22:49:01.593106 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585564    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 22:49:01.593106 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585567    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 22:49:01.593106 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585569    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 22:49:01.593106 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585579    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 22:49:01.593106 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585582    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 22:49:01.593106 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585584    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 22:49:01.593106 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585586    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 22:49:01.593106 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585589    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 22:49:01.593106 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585591    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 22:49:01.593106 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585594    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 22:49:01.593573 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585596    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 22:49:01.593573 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585599    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 22:49:01.593573 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585601    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 22:49:01.593573 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585604    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 22:49:01.593573 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585606    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 22:49:01.593573 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585608    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 22:49:01.593573 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585611    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 22:49:01.593573 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585613    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 22:49:01.593573 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585615    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 22:49:01.593573 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585618    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 22:49:01.593573 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585620    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 22:49:01.593573 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585623    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 22:49:01.593573 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585625    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 22:49:01.593573 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585627    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 22:49:01.593573 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585630    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 22:49:01.593573 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585632    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 22:49:01.593573 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585635    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 22:49:01.593573 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585637    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 22:49:01.593573 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585639    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 22:49:01.593573 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585642    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 22:49:01.594060 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585644    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 22:49:01.594060 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585647    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 22:49:01.594060 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585649    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 22:49:01.594060 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585653    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 22:49:01.594060 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585656    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 22:49:01.594060 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585658    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 22:49:01.594060 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585666    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 22:49:01.594060 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585669    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 22:49:01.594060 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585671    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 22:49:01.594060 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585674    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 22:49:01.594060 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585676    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 22:49:01.594060 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585679    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 22:49:01.594060 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585681    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 22:49:01.594060 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585684    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 22:49:01.594060 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585686    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 22:49:01.594060 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585689    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 22:49:01.594060 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585693    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 22:49:01.594060 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585696    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 22:49:01.594060 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585699    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 22:49:01.594511 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585702    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 22:49:01.594511 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585704    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 22:49:01.594511 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585707    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 22:49:01.594511 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585709    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 22:49:01.594511 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585712    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 22:49:01.594511 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585714    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 22:49:01.594511 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585716    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 22:49:01.594511 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585719    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 22:49:01.594511 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585721    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 22:49:01.594511 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585723    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 22:49:01.594511 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585726    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 22:49:01.594511 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585728    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 22:49:01.594511 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585731    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 22:49:01.594511 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585733    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 22:49:01.594511 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585735    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 22:49:01.594511 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585738    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 22:49:01.594511 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585742    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 22:49:01.594511 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585744    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 22:49:01.594511 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585747    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 22:49:01.594511 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585750    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 22:49:01.594998 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585758    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 22:49:01.594998 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585761    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 22:49:01.594998 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585763    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 22:49:01.594998 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585766    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 22:49:01.594998 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585768    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 22:49:01.594998 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585771    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 22:49:01.594998 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585774    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 22:49:01.594998 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585777    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 22:49:01.594998 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585779    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 22:49:01.594998 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585782    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 22:49:01.594998 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585784    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 22:49:01.594998 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585786    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 22:49:01.594998 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585789    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 22:49:01.594998 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585791    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 22:49:01.594998 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585794    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 22:49:01.594998 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585796    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 22:49:01.594998 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585798    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 22:49:01.594998 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585801    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 22:49:01.594998 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585803    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 22:49:01.594998 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585806    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 22:49:01.595520 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585813    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 22:49:01.595520 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585817    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 22:49:01.595520 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585820    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 22:49:01.595520 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585824    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 22:49:01.595520 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585827    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 22:49:01.595520 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585829    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 22:49:01.595520 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585831    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 22:49:01.595520 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585834    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 22:49:01.595520 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585838    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 22:49:01.595520 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585841    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 22:49:01.595520 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585843    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 22:49:01.595520 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:01.585846    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 22:49:01.595520 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.585904    2575 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 22:49:01.595520 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.588250    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 22:49:01.595520 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.589278    2575 server.go:1014] "Starting client certificate rotation"
Oct 31 22:49:01.595520 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.589396    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 22:49:01.595520 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.589449    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 22:49:01.595926 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.593935    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 22:49:01.595926 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.595030    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 22:49:01.607582 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.607562    2575 log.go:25] "Validated CRI v1 runtime API"
Oct 31 22:49:01.610616 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.610604    2575 log.go:25] "Validated CRI v1 image API"
Oct 31 22:49:01.611470 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.611443    2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 22:49:01.619459 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.619439    2575 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 aab09ba8-6576-46e8-af39-036eb8136f00:/dev/nvme0n1p4 f9f5192e-056d-4fe0-9a3c-378434ef4a61:/dev/nvme0n1p3]
Oct 31 22:49:01.619562 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.619458    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 22:49:01.625598 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.625476    2575 manager.go:217] Machine: {Timestamp:2026-10-31 22:49:01.623977655 +0000 UTC m=+0.382424091 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3161016 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27926e91027f103fd8d11211f42a5f SystemUUID:ec27926e-9102-7f10-3fd8-d11211f42a5f BootID:79095032-3b68-469b-9072-952d60400029 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:da:ca:ad:07 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:da:ca:ad:07 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:a8:2f:e8:99:b5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 22:49:01.625598 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.625588    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 22:49:01.625717 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.625671    2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 22:49:01.629585 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.629567    2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 22:49:01.629958 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.629934    2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 22:49:01.630099 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.629962    2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-163.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 22:49:01.630180 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.630105    2575 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 22:49:01.630180 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.630112    2575 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 22:49:01.630180 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.630126    2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 22:49:01.630180 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.630140    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 22:49:01.631599 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.631589    2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 22:49:01.635736 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.635723    2575 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 22:49:01.635847 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.635838    2575 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 22:49:01.635883 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.635869    2575 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 22:49:01.635883 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.635880    2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 22:49:01.635936 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.635889    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 22:49:01.635936 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.635898    2575 kubelet.go:405] "Adding apiserver pod source"
Oct 31 22:49:01.635936 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.635907    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 22:49:01.636306 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.636289    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 22:49:01.638833 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.638816    2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 22:49:01.646318 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.646302    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 22:49:01.647201 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.647187    2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 22:49:01.647269 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.647242    2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 22:49:01.647439 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.647428    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 22:49:01.647490 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.647443    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 22:49:01.647490 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.647450    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 22:49:01.647490 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.647455    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 22:49:01.647490 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.647461    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 22:49:01.647490 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.647467    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 22:49:01.648081 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.648073    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 22:49:01.648109 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.648082    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 22:49:01.648109 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.648090    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 22:49:01.648109 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.648096    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 22:49:01.648109 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.648104    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 22:49:01.648230 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.648112    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 22:49:01.648929 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.648921    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 22:49:01.648962 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.648930    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 22:49:01.651200 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.651185    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q7pfw"
Oct 31 22:49:01.652811 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.652791    2575 server.go:1263] "Started kubelet"
Oct 31 22:49:01.652899 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.652867    2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 22:49:01.653010 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.652963    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 22:49:01.653084 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.653019    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 22:49:01.653624 ip-10-0-135-163 systemd[1]: Started Kubernetes Kubelet.
Oct 31 22:49:01.655145 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.654664    2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 22:49:01.655145 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.655602    2575 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 22:49:01.655145 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.656137    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-163.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 22:49:01.661897 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:01.661875    2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 22:49:01.662151 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.662131    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 22:49:01.663844 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.663583    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q7pfw"
Oct 31 22:49:01.664102 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.664084    2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 22:49:01.664599 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:01.664579    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-163.us-east-2.compute.internal\" not found"
Oct 31 22:49:01.664837 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.664820    2575 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 22:49:01.664880 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.664842    2575 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 22:49:01.665011 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.664997    2575 factory.go:55] Registering systemd factory
Oct 31 22:49:01.665071 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.665018    2575 factory.go:223] Registration of the systemd container factory successfully
Oct 31 22:49:01.665116 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.665105    2575 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 22:49:01.665143 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.665117    2575 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 22:49:01.665298 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.665247    2575 factory.go:153] Registering CRI-O factory
Oct 31 22:49:01.665298 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.665265    2575 factory.go:223] Registration of the crio container factory successfully
Oct 31 22:49:01.665442 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.665331    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 22:49:01.665442 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.665352    2575 factory.go:103] Registering Raw factory
Oct 31 22:49:01.665442 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.665390    2575 manager.go:1199] Started watching for new ooms in manager
Oct 31 22:49:01.665621 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.665586    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 22:49:01.666368 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.666339    2575 manager.go:319] Starting recovery of all containers
Oct 31 22:49:01.667088 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:01.667066    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-163.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 31 22:49:01.667088 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:01.667070    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 31 22:49:01.668271 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:01.667145    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-163.us-east-2.compute.internal.18e3bedf0283d09c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-163.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-163.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 22:49:01.652766876 +0000 UTC m=+0.411213310,LastTimestamp:2026-10-31 22:49:01.652766876 +0000 UTC m=+0.411213310,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-163.us-east-2.compute.internal,}"
Oct 31 22:49:01.677624 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:01.670694    2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-163.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 31 22:49:01.678093 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.678057    2575 manager.go:324] Recovery completed
Oct 31 22:49:01.678937 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.678914    2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 22:49:01.684531 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.684516    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 22:49:01.686120 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.686105    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-163.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 22:49:01.686191 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.686131    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-163.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 22:49:01.686191 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.686141    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-163.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 22:49:01.686734 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.686719    2575 cpu_manager.go:226] "Starting" policy="none"
Oct 31 22:49:01.686734 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.686728    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 22:49:01.686734 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.686732    2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 22:49:01.686907 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.686751    2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 22:49:01.687527 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.687509    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-163.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 22:49:01.687609 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.687547    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-163.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 22:49:01.687609 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.687560    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-163.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 22:49:01.688830 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.688819    2575 policy_none.go:50] "Start"
Oct 31 22:49:01.688872 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.688834    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 22:49:01.688872 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.688843    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 22:49:01.690866 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.690851    2575 policy_none.go:44] "Start"
Oct 31 22:49:01.743489 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.735914    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 22:49:01.743489 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.737233    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 22:49:01.743489 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.737251    2575 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 22:49:01.743489 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.737268    2575 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 22:49:01.743489 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:01.737307    2575 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 22:49:01.743489 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.738154    2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 22:49:01.743489 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.741716    2575 manager.go:341] "Starting Device Plugin manager"
Oct 31 22:49:01.743489 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:01.741807    2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 22:49:01.743489 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.741818    2575 server.go:87] "Starting device plugin registration server"
Oct 31 22:49:01.743489 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.742094    2575 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 22:49:01.743489 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.742105    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 22:49:01.743489 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.742202    2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 22:49:01.743489 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.742284    2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 22:49:01.743489 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.742293    2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 22:49:01.743489 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.742430    2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 22:49:01.744218 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:01.743573    2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 22:49:01.744218 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:01.743615    2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-163.us-east-2.compute.internal\" not found"
Oct 31 22:49:01.837646 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.837561    2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-163.us-east-2.compute.internal"]
Oct 31 22:49:01.837646 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.837641    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 22:49:01.838694 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.838670    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-163.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 22:49:01.838805 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.838704    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-163.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 22:49:01.838805 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.838719    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-163.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 22:49:01.840765 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.840752    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 22:49:01.840909 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.840894    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:01.840948 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.840925    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 22:49:01.841504 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.841487    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-163.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 22:49:01.841583 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.841512    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-163.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 22:49:01.841583 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.841522    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-163.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 22:49:01.841583 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.841489    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-163.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 22:49:01.841684 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.841594    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-163.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 22:49:01.841684 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.841614    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-163.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 22:49:01.842227 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.842213    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 22:49:01.842915 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.842900    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-163.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 22:49:01.843001 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.842928    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-163.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 22:49:01.843001 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.842943    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-163.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 22:49:01.843001 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.842968    2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:01.843609 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.843594    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:01.843659 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.843631    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 22:49:01.844274 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.844261    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-163.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 22:49:01.844348 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.844291    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-163.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 22:49:01.844348 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.844305    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-163.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 22:49:01.852974 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.852952    2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:01.853075 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:01.853002    2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-163.us-east-2.compute.internal\": node \"ip-10-0-135-163.us-east-2.compute.internal\" not found"
Oct 31 22:49:01.871743 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:01.871722    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-163.us-east-2.compute.internal\" not found"
Oct 31 22:49:01.881765 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:01.881748    2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-163.us-east-2.compute.internal\" not found" node="ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:01.883650 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:01.883635    2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-163.us-east-2.compute.internal\" not found" node="ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:01.966233 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.966204    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8ec0c93a8e98312f4e27fee411631fef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal\" (UID: \"8ec0c93a8e98312f4e27fee411631fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:01.966390 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.966235    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ec0c93a8e98312f4e27fee411631fef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal\" (UID: \"8ec0c93a8e98312f4e27fee411631fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:01.966390 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:01.966259    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2a808dc7f3e00ce75e0f6c4e3f550fac-config\") pod \"kube-apiserver-proxy-ip-10-0-135-163.us-east-2.compute.internal\" (UID: \"2a808dc7f3e00ce75e0f6c4e3f550fac\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:01.972317 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:01.972296    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-163.us-east-2.compute.internal\" not found"
Oct 31 22:49:02.066480 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:02.066438    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8ec0c93a8e98312f4e27fee411631fef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal\" (UID: \"8ec0c93a8e98312f4e27fee411631fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:02.066480 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:02.066483    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ec0c93a8e98312f4e27fee411631fef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal\" (UID: \"8ec0c93a8e98312f4e27fee411631fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:02.066647 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:02.066524    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8ec0c93a8e98312f4e27fee411631fef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal\" (UID: \"8ec0c93a8e98312f4e27fee411631fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:02.066647 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:02.066524    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2a808dc7f3e00ce75e0f6c4e3f550fac-config\") pod \"kube-apiserver-proxy-ip-10-0-135-163.us-east-2.compute.internal\" (UID: \"2a808dc7f3e00ce75e0f6c4e3f550fac\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:02.066647 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:02.066560    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2a808dc7f3e00ce75e0f6c4e3f550fac-config\") pod \"kube-apiserver-proxy-ip-10-0-135-163.us-east-2.compute.internal\" (UID: \"2a808dc7f3e00ce75e0f6c4e3f550fac\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:02.066647 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:02.066562    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ec0c93a8e98312f4e27fee411631fef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal\" (UID: \"8ec0c93a8e98312f4e27fee411631fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:02.072569 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:02.072548    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-163.us-east-2.compute.internal\" not found"
Oct 31 22:49:02.173227 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:02.173191    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-163.us-east-2.compute.internal\" not found"
Oct 31 22:49:02.186395 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:02.186352    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:02.188493 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:02.188462    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:02.274170 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:02.274002    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-163.us-east-2.compute.internal\" not found"
Oct 31 22:49:02.374424 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:02.374398    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-163.us-east-2.compute.internal\" not found"
Oct 31 22:49:02.474883 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:02.474818    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-163.us-east-2.compute.internal\" not found"
Oct 31 22:49:02.575204 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:02.575183    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-163.us-east-2.compute.internal\" not found"
Oct 31 22:49:02.589348 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:02.589327    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 22:49:02.589513 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:02.589463    2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 22:49:02.589561 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:02.589512    2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 22:49:02.670025 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:02.669965    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 22:44:01 +0000 UTC" deadline="2028-05-03 21:58:17.891986511 +0000 UTC"
Oct 31 22:49:02.670025 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:02.670016    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13199h9m15.221973595s"
Oct 31 22:49:02.675514 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:02.675491    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-163.us-east-2.compute.internal\" not found"
Oct 31 22:49:02.768820 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:02.768788    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec0c93a8e98312f4e27fee411631fef.slice/crio-06263e2bcd2200a9857b0e6aa89953bc1872c669e9ed1bd071d6c7dac6538c17 WatchSource:0}: Error finding container 06263e2bcd2200a9857b0e6aa89953bc1872c669e9ed1bd071d6c7dac6538c17: Status 404 returned error can't find the container with id 06263e2bcd2200a9857b0e6aa89953bc1872c669e9ed1bd071d6c7dac6538c17
Oct 31 22:49:02.769198 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:02.769170    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a808dc7f3e00ce75e0f6c4e3f550fac.slice/crio-fa7a63b33e9aec9debd129fc805777e20079a37d22133f1983458413291498d9 WatchSource:0}: Error finding container fa7a63b33e9aec9debd129fc805777e20079a37d22133f1983458413291498d9: Status 404 returned error can't find the container with id fa7a63b33e9aec9debd129fc805777e20079a37d22133f1983458413291498d9
Oct 31 22:49:02.775497 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:02.775478    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 22:49:02.775584 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:02.775547    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-163.us-east-2.compute.internal\" not found"
Oct 31 22:49:02.862334 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:02.862307    2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 22:49:02.865548 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:02.865532    2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:02.866873 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:02.866858    2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 22:49:02.878515 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:02.878491    2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 22:49:02.879605 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:02.879591    2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:02.889064 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:02.889039    2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 22:49:03.401611 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.401583    2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 22:49:03.636934 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.636881    2575 apiserver.go:52] "Watching apiserver"
Oct 31 22:49:03.663044 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.662980    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 22:49:03.679229 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.679203    2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 22:49:03.703233 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.702444    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-x5fr7"
Oct 31 22:49:03.703233 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.702686    2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 22:49:03.703233 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.702968    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-163.us-east-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-7s9nl","openshift-cluster-node-tuning-operator/tuned-dcczv","openshift-multus/multus-7pksb","kube-system/global-pull-secret-syncer-qv88p","openshift-multus/multus-additional-cni-plugins-2s8p8","openshift-network-operator/iptables-alerter-lnhbm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2","openshift-dns/node-resolver-cn8jx","openshift-image-registry/node-ca-xw8kl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal","openshift-multus/network-metrics-daemon-5qkg4","openshift-network-diagnostics/network-check-target-ddxkp","kube-system/konnectivity-agent-vc2j7"]
Oct 31 22:49:03.705612 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.705592    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lnhbm"
Oct 31 22:49:03.707402 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.707382    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddxkp"
Oct 31 22:49:03.707497 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:03.707451    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddxkp" podUID="efec9075-8e74-475d-8160-48e356140743"
Oct 31 22:49:03.709394 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.709375    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.711882 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.711863    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.714388 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.714372    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:03.714472 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:03.714428    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv88p" podUID="620515c1-daf9-48e9-9718-a71057b3b615"
Oct 31 22:49:03.715677 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.715651    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 22:49:03.715677 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.715655    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 22:49:03.715774 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.715660    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pn67g\""
Oct 31 22:49:03.715774 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.715701    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 22:49:03.717133 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.717117    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:03.717653 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.717636    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dkn4d\""
Oct 31 22:49:03.718586 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.718568    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cn8jx"
Oct 31 22:49:03.722764 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.722745    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vc2j7"
Oct 31 22:49:03.723184 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.723126    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 22:49:03.723184 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.723149    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-x5fr7"
Oct 31 22:49:03.726351 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.726327    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.728552 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.728533    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xw8kl"
Oct 31 22:49:03.730669 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.730652    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:03.733164 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.733147    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:03.733249 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:03.733212    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qkg4" podUID="81cc6a3e-e646-4287-84a7-d6830c40784b"
Oct 31 22:49:03.744200 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.744141    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-163.us-east-2.compute.internal" event={"ID":"2a808dc7f3e00ce75e0f6c4e3f550fac","Type":"ContainerStarted","Data":"fa7a63b33e9aec9debd129fc805777e20079a37d22133f1983458413291498d9"}
Oct 31 22:49:03.746610 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.746585    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal" event={"ID":"8ec0c93a8e98312f4e27fee411631fef","Type":"ContainerStarted","Data":"06263e2bcd2200a9857b0e6aa89953bc1872c669e9ed1bd071d6c7dac6538c17"}
Oct 31 22:49:03.766723 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.766702    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 22:49:03.775053 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775015    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-modprobe-d\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.775149 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775065    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-var-lib-kubelet\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.775149 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775086    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-run-netns\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.775253 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775141    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79xfm\" (UniqueName: \"kubernetes.io/projected/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-kube-api-access-79xfm\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.775253 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775190    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-device-dir\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:03.775253 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775218    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-cnibin\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.775253 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775243    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/620515c1-daf9-48e9-9718-a71057b3b615-original-pull-secret\") pod \"global-pull-secret-syncer-qv88p\" (UID: \"620515c1-daf9-48e9-9718-a71057b3b615\") " pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:03.775478 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775279    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e46ea70b-37a4-4934-920c-ecdab212ecc6-hosts-file\") pod \"node-resolver-cn8jx\" (UID: \"e46ea70b-37a4-4934-920c-ecdab212ecc6\") " pod="openshift-dns/node-resolver-cn8jx"
Oct 31 22:49:03.775478 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775373    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-kubelet\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.775478 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775402    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-run-ovn\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.775478 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775425    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-system-cni-dir\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.775478 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775445    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-var-lib-kubelet\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.775478 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775468    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-multus-daemon-config\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.775675 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775489    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b1db31b-aec0-495b-8b0c-4abbde3acb28-host\") pod \"node-ca-xw8kl\" (UID: \"8b1db31b-aec0-495b-8b0c-4abbde3acb28\") " pod="openshift-image-registry/node-ca-xw8kl"
Oct 31 22:49:03.775675 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775510    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-run-k8s-cni-cncf-io\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.775675 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775531    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-multus-conf-dir\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.775675 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775579    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/76488729-53ad-4536-8157-7d41ef49863b-cnibin\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:03.775675 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775603    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-etc-openvswitch\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.775675 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775626    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cxsk\" (UniqueName: \"kubernetes.io/projected/81cc6a3e-e646-4287-84a7-d6830c40784b-kube-api-access-7cxsk\") pod \"network-metrics-daemon-5qkg4\" (UID: \"81cc6a3e-e646-4287-84a7-d6830c40784b\") " pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:03.775675 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775649    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4sx8\" (UniqueName: \"kubernetes.io/projected/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-kube-api-access-r4sx8\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.775675 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775670    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8b1db31b-aec0-495b-8b0c-4abbde3acb28-serviceca\") pod \"node-ca-xw8kl\" (UID: \"8b1db31b-aec0-495b-8b0c-4abbde3acb28\") " pod="openshift-image-registry/node-ca-xw8kl"
Oct 31 22:49:03.775911 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775691    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-socket-dir\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:03.775911 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775707    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-etc-selinux\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:03.775911 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775720    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-sysctl-d\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.775911 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775736    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/76488729-53ad-4536-8157-7d41ef49863b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:03.775911 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775774    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-sysctl-conf\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.775911 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775788    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-systemd\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.775911 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775811    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-run\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.775911 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775831    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-etc-kubernetes\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.775911 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775848    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-cni-netd\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.775911 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775860    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-systemd-units\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.775911 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775880    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65821d7f-a625-4696-9908-44a8eb6ecdde-host-slash\") pod \"iptables-alerter-lnhbm\" (UID: \"65821d7f-a625-4696-9908-44a8eb6ecdde\") " pod="openshift-network-operator/iptables-alerter-lnhbm"
Oct 31 22:49:03.775911 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775892    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-kubernetes\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.775911 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775903    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-host\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.776249 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775926    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76488729-53ad-4536-8157-7d41ef49863b-system-cni-dir\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:03.776249 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775968    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76488729-53ad-4536-8157-7d41ef49863b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:03.776249 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.775990    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-ovn-node-metrics-cert\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.776249 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776013    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-registration-dir\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:03.776249 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776042    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/76488729-53ad-4536-8157-7d41ef49863b-cni-binary-copy\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:03.776249 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776061    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qndh\" (UniqueName: \"kubernetes.io/projected/e46ea70b-37a4-4934-920c-ecdab212ecc6-kube-api-access-5qndh\") pod \"node-resolver-cn8jx\" (UID: \"e46ea70b-37a4-4934-920c-ecdab212ecc6\") " pod="openshift-dns/node-resolver-cn8jx"
Oct 31 22:49:03.776249 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776075    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hqz8\" (UniqueName: \"kubernetes.io/projected/efec9075-8e74-475d-8160-48e356140743-kube-api-access-9hqz8\") pod \"network-check-target-ddxkp\" (UID: \"efec9075-8e74-475d-8160-48e356140743\") " pod="openshift-network-diagnostics/network-check-target-ddxkp"
Oct 31 22:49:03.776249 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776087    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-run-openvswitch\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.776249 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776104    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/546c0de6-9661-41cf-a068-cbbf88d05c2e-tmp\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.776249 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776117    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-multus-cni-dir\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.776249 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776135    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/620515c1-daf9-48e9-9718-a71057b3b615-dbus\") pod \"global-pull-secret-syncer-qv88p\" (UID: \"620515c1-daf9-48e9-9718-a71057b3b615\") " pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:03.776249 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776168    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-sys-fs\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:03.776249 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776181    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81cc6a3e-e646-4287-84a7-d6830c40784b-metrics-certs\") pod \"network-metrics-daemon-5qkg4\" (UID: \"81cc6a3e-e646-4287-84a7-d6830c40784b\") " pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:03.776249 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776208    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-tuned\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.776249 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776224    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-hostroot\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.776249 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776242    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rwp2\" (UniqueName: \"kubernetes.io/projected/76488729-53ad-4536-8157-7d41ef49863b-kube-api-access-9rwp2\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:03.776710 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776257    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-node-log\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.776710 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776282    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/620515c1-daf9-48e9-9718-a71057b3b615-kubelet-config\") pod \"global-pull-secret-syncer-qv88p\" (UID: \"620515c1-daf9-48e9-9718-a71057b3b615\") " pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:03.776710 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776296    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-log-socket\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.776710 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776334    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.776710 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776374    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65821d7f-a625-4696-9908-44a8eb6ecdde-iptables-alerter-script\") pod \"iptables-alerter-lnhbm\" (UID: \"65821d7f-a625-4696-9908-44a8eb6ecdde\") " pod="openshift-network-operator/iptables-alerter-lnhbm"
Oct 31 22:49:03.776710 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776396    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bz55\" (UniqueName: \"kubernetes.io/projected/546c0de6-9661-41cf-a068-cbbf88d05c2e-kube-api-access-4bz55\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.776710 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776420    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-var-lib-cni-bin\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.776710 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776440    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5f102a31-27f2-410b-9e36-2cdb8d9e852c-konnectivity-ca\") pod \"konnectivity-agent-vc2j7\" (UID: \"5f102a31-27f2-410b-9e36-2cdb8d9e852c\") " pod="kube-system/konnectivity-agent-vc2j7"
Oct 31 22:49:03.776710 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776466    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-slash\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.776710 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776479    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-env-overrides\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.776710 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776493    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:03.776710 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776515    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spqnp\" (UniqueName: \"kubernetes.io/projected/a2ea746c-d257-4b00-be3e-ba726bd549de-kube-api-access-spqnp\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:03.776710 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776528    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz9d9\" (UniqueName: \"kubernetes.io/projected/65821d7f-a625-4696-9908-44a8eb6ecdde-kube-api-access-mz9d9\") pod \"iptables-alerter-lnhbm\" (UID: \"65821d7f-a625-4696-9908-44a8eb6ecdde\") " pod="openshift-network-operator/iptables-alerter-lnhbm"
Oct 31 22:49:03.776710 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776540    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-lib-modules\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.776710 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776562    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-multus-socket-dir-parent\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.776710 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776582    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-run-ovn-kubernetes\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.777163 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776604    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-ovnkube-config\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.777163 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776627    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdr9n\" (UniqueName: \"kubernetes.io/projected/8b1db31b-aec0-495b-8b0c-4abbde3acb28-kube-api-access-hdr9n\") pod \"node-ca-xw8kl\" (UID: \"8b1db31b-aec0-495b-8b0c-4abbde3acb28\") " pod="openshift-image-registry/node-ca-xw8kl"
Oct 31 22:49:03.777163 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776643    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-run-netns\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.777163 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776657    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-sysconfig\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.777163 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776670    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-sys\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.777163 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776684    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/76488729-53ad-4536-8157-7d41ef49863b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:03.777163 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776698    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5f102a31-27f2-410b-9e36-2cdb8d9e852c-agent-certs\") pod \"konnectivity-agent-vc2j7\" (UID: \"5f102a31-27f2-410b-9e36-2cdb8d9e852c\") " pod="kube-system/konnectivity-agent-vc2j7"
Oct 31 22:49:03.777163 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776713    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-cni-bin\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.777163 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776732    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-ovnkube-script-lib\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.777163 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776754    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-cni-binary-copy\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.777163 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776775    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-var-lib-cni-multus\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.777163 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776790    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-run-multus-certs\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.777163 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776803    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e46ea70b-37a4-4934-920c-ecdab212ecc6-tmp-dir\") pod \"node-resolver-cn8jx\" (UID: \"e46ea70b-37a4-4934-920c-ecdab212ecc6\") " pod="openshift-dns/node-resolver-cn8jx"
Oct 31 22:49:03.777163 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776815    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-run-systemd\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.777163 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.776834    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-var-lib-openvswitch\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.806715 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.806686    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 22:49:03.806865 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.806728    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2drsv\""
Oct 31 22:49:03.808612 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.808594    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gr9dz\""
Oct 31 22:49:03.809099 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.809071    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-2gb2z\""
Oct 31 22:49:03.809099 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.809078    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-l6qjt\""
Oct 31 22:49:03.809243 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.809097    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 22:49:03.809243 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.809179    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gdsw9\""
Oct 31 22:49:03.809339 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.809296    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kg7vs\""
Oct 31 22:49:03.809408 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.809346    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-j2f6b\""
Oct 31 22:49:03.809451 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.809428    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 22:49:03.809451 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.809435    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 22:49:03.810784 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.810767    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 22:49:03.810867 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.810792    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 22:49:03.810867 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.810830    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 22:49:03.811200 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.811183    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 22:49:03.811279 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.811217    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 22:49:03.811427 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.811409    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 22:49:03.811559 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.811501    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 22:49:03.811559 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.811541    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 22:49:03.811676 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.811639    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 22:49:03.811725 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.811675    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 22:49:03.811725 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.811686    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 22:49:03.811725 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.811718    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 22:49:03.811849 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.811786    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 22:49:03.811849 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.811800    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 22:49:03.812178 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.812161    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 22:49:03.812219 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.812198    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 22:49:03.812260 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.812242    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 22:49:03.812345 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.812307    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 22:49:03.813274 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.813259    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 22:49:03.877500 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.877465    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-sysctl-d\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.877668 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.877512    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/76488729-53ad-4536-8157-7d41ef49863b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:03.877668 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.877539    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-sysctl-conf\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.877668 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.877554    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-systemd\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.877668 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.877567    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-run\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.877668 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.877595    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-etc-kubernetes\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.877668 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.877611    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-cni-netd\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.877668 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.877626    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-systemd-units\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.877668 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.877646    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65821d7f-a625-4696-9908-44a8eb6ecdde-host-slash\") pod \"iptables-alerter-lnhbm\" (UID: \"65821d7f-a625-4696-9908-44a8eb6ecdde\") " pod="openshift-network-operator/iptables-alerter-lnhbm"
Oct 31 22:49:03.877668 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.877667    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-kubernetes\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.877951 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.877683    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-host\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.877951 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.877696    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76488729-53ad-4536-8157-7d41ef49863b-system-cni-dir\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:03.877951 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.877709    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76488729-53ad-4536-8157-7d41ef49863b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:03.877951 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.877726    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-ovn-node-metrics-cert\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.877951 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.877744    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-registration-dir\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:03.877951 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.877759    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/76488729-53ad-4536-8157-7d41ef49863b-cni-binary-copy\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:03.877951 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.877773    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qndh\" (UniqueName: \"kubernetes.io/projected/e46ea70b-37a4-4934-920c-ecdab212ecc6-kube-api-access-5qndh\") pod \"node-resolver-cn8jx\" (UID: \"e46ea70b-37a4-4934-920c-ecdab212ecc6\") " pod="openshift-dns/node-resolver-cn8jx"
Oct 31 22:49:03.877951 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.877787    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hqz8\" (UniqueName: \"kubernetes.io/projected/efec9075-8e74-475d-8160-48e356140743-kube-api-access-9hqz8\") pod \"network-check-target-ddxkp\" (UID: \"efec9075-8e74-475d-8160-48e356140743\") " pod="openshift-network-diagnostics/network-check-target-ddxkp"
Oct 31 22:49:03.877951 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.877804    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-run-openvswitch\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.877951 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.877822    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/546c0de6-9661-41cf-a068-cbbf88d05c2e-tmp\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.877951 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.877836    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-multus-cni-dir\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.877951 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.877848    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/620515c1-daf9-48e9-9718-a71057b3b615-dbus\") pod \"global-pull-secret-syncer-qv88p\" (UID: \"620515c1-daf9-48e9-9718-a71057b3b615\") " pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:03.877951 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.877861    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-sys-fs\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:03.877951 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.877874    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81cc6a3e-e646-4287-84a7-d6830c40784b-metrics-certs\") pod \"network-metrics-daemon-5qkg4\" (UID: \"81cc6a3e-e646-4287-84a7-d6830c40784b\") " pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:03.877951 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.877892    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-tuned\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.877951 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.877907    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-hostroot\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.877951 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.877919    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rwp2\" (UniqueName: \"kubernetes.io/projected/76488729-53ad-4536-8157-7d41ef49863b-kube-api-access-9rwp2\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:03.878481 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.877932    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-node-log\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.878481 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.877946    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/620515c1-daf9-48e9-9718-a71057b3b615-kubelet-config\") pod \"global-pull-secret-syncer-qv88p\" (UID: \"620515c1-daf9-48e9-9718-a71057b3b615\") " pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:03.878481 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.877967    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-log-socket\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.878481 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.877982    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.878481 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.877996    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65821d7f-a625-4696-9908-44a8eb6ecdde-iptables-alerter-script\") pod \"iptables-alerter-lnhbm\" (UID: \"65821d7f-a625-4696-9908-44a8eb6ecdde\") " pod="openshift-network-operator/iptables-alerter-lnhbm"
Oct 31 22:49:03.878481 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878009    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bz55\" (UniqueName: \"kubernetes.io/projected/546c0de6-9661-41cf-a068-cbbf88d05c2e-kube-api-access-4bz55\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.878481 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878026    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-var-lib-cni-bin\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.878481 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878045    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5f102a31-27f2-410b-9e36-2cdb8d9e852c-konnectivity-ca\") pod \"konnectivity-agent-vc2j7\" (UID: \"5f102a31-27f2-410b-9e36-2cdb8d9e852c\") " pod="kube-system/konnectivity-agent-vc2j7"
Oct 31 22:49:03.878481 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878062    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-slash\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.878481 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878075    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-env-overrides\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.878481 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878088    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:03.878481 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878109    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spqnp\" (UniqueName: \"kubernetes.io/projected/a2ea746c-d257-4b00-be3e-ba726bd549de-kube-api-access-spqnp\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:03.878481 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878126    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz9d9\" (UniqueName: \"kubernetes.io/projected/65821d7f-a625-4696-9908-44a8eb6ecdde-kube-api-access-mz9d9\") pod \"iptables-alerter-lnhbm\" (UID: \"65821d7f-a625-4696-9908-44a8eb6ecdde\") " pod="openshift-network-operator/iptables-alerter-lnhbm"
Oct 31 22:49:03.878481 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878138    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-lib-modules\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.878481 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878152    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-multus-socket-dir-parent\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.878481 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878165    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-run-ovn-kubernetes\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.878481 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878185    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-ovnkube-config\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.879150 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878200    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdr9n\" (UniqueName: \"kubernetes.io/projected/8b1db31b-aec0-495b-8b0c-4abbde3acb28-kube-api-access-hdr9n\") pod \"node-ca-xw8kl\" (UID: \"8b1db31b-aec0-495b-8b0c-4abbde3acb28\") " pod="openshift-image-registry/node-ca-xw8kl"
Oct 31 22:49:03.879150 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878213    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-run-netns\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.879150 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878226    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-sysconfig\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.879150 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878243    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-sys\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.879150 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878262    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/76488729-53ad-4536-8157-7d41ef49863b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:03.879150 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878281    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5f102a31-27f2-410b-9e36-2cdb8d9e852c-agent-certs\") pod \"konnectivity-agent-vc2j7\" (UID: \"5f102a31-27f2-410b-9e36-2cdb8d9e852c\") " pod="kube-system/konnectivity-agent-vc2j7"
Oct 31 22:49:03.879150 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878294    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-cni-bin\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.879150 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878307    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-ovnkube-script-lib\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.879150 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878320    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-cni-binary-copy\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.879150 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878339    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-var-lib-cni-multus\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.879150 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878370    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-run-multus-certs\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.879150 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878402    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e46ea70b-37a4-4934-920c-ecdab212ecc6-tmp-dir\") pod \"node-resolver-cn8jx\" (UID: \"e46ea70b-37a4-4934-920c-ecdab212ecc6\") " pod="openshift-dns/node-resolver-cn8jx"
Oct 31 22:49:03.879150 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878428    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-run-systemd\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.879150 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878453    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-var-lib-openvswitch\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.879150 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878479    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-modprobe-d\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.879150 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878502    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-var-lib-kubelet\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.879150 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878522    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-run-netns\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.879928 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878544    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79xfm\" (UniqueName: \"kubernetes.io/projected/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-kube-api-access-79xfm\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.879928 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878568    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-device-dir\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:03.879928 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878592    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-cnibin\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.879928 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878615    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/620515c1-daf9-48e9-9718-a71057b3b615-original-pull-secret\") pod \"global-pull-secret-syncer-qv88p\" (UID: \"620515c1-daf9-48e9-9718-a71057b3b615\") " pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:03.879928 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878638    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e46ea70b-37a4-4934-920c-ecdab212ecc6-hosts-file\") pod \"node-resolver-cn8jx\" (UID: \"e46ea70b-37a4-4934-920c-ecdab212ecc6\") " pod="openshift-dns/node-resolver-cn8jx"
Oct 31 22:49:03.879928 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878659    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-kubelet\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.879928 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878686    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-run-ovn\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.879928 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878708    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-system-cni-dir\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.879928 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878730    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-var-lib-kubelet\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.879928 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878754    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-multus-daemon-config\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.879928 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878779    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b1db31b-aec0-495b-8b0c-4abbde3acb28-host\") pod \"node-ca-xw8kl\" (UID: \"8b1db31b-aec0-495b-8b0c-4abbde3acb28\") " pod="openshift-image-registry/node-ca-xw8kl"
Oct 31 22:49:03.879928 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878804    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-run-k8s-cni-cncf-io\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.879928 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878823    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-multus-conf-dir\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.879928 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878838    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/76488729-53ad-4536-8157-7d41ef49863b-cnibin\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:03.879928 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878853    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-etc-openvswitch\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.879928 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878873    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cxsk\" (UniqueName: \"kubernetes.io/projected/81cc6a3e-e646-4287-84a7-d6830c40784b-kube-api-access-7cxsk\") pod \"network-metrics-daemon-5qkg4\" (UID: \"81cc6a3e-e646-4287-84a7-d6830c40784b\") " pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:03.879928 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878900    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4sx8\" (UniqueName: \"kubernetes.io/projected/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-kube-api-access-r4sx8\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.879928 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878917    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8b1db31b-aec0-495b-8b0c-4abbde3acb28-serviceca\") pod \"node-ca-xw8kl\" (UID: \"8b1db31b-aec0-495b-8b0c-4abbde3acb28\") " pod="openshift-image-registry/node-ca-xw8kl"
Oct 31 22:49:03.880684 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878937    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-socket-dir\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:03.880684 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.878959    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-etc-selinux\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:03.880684 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.879139    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-etc-selinux\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:03.880684 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.879245    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-sysctl-d\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.880684 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.879495    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-lib-modules\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.880684 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.879510    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-multus-cni-dir\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.880684 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.879572    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/620515c1-daf9-48e9-9718-a71057b3b615-dbus\") pod \"global-pull-secret-syncer-qv88p\" (UID: \"620515c1-daf9-48e9-9718-a71057b3b615\") " pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:03.880684 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.879623    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-sys-fs\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:03.880684 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.879656    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-sysctl-conf\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.880684 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.879706    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-systemd\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.880684 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:03.879710    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 22:49:03.880684 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.879750    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-run\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.880684 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:03.879782    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81cc6a3e-e646-4287-84a7-d6830c40784b-metrics-certs podName:81cc6a3e-e646-4287-84a7-d6830c40784b nodeName:}" failed. No retries permitted until 2026-10-31 22:49:04.379747524 +0000 UTC m=+3.138193965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81cc6a3e-e646-4287-84a7-d6830c40784b-metrics-certs") pod "network-metrics-daemon-5qkg4" (UID: "81cc6a3e-e646-4287-84a7-d6830c40784b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 22:49:03.880684 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.879790    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-etc-kubernetes\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.880684 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.879820    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-multus-socket-dir-parent\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.880684 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.879828    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-cni-netd\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.880684 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.879840    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-hostroot\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.881374 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.879863    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-run-ovn-kubernetes\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.881374 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.879872    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65821d7f-a625-4696-9908-44a8eb6ecdde-host-slash\") pod \"iptables-alerter-lnhbm\" (UID: \"65821d7f-a625-4696-9908-44a8eb6ecdde\") " pod="openshift-network-operator/iptables-alerter-lnhbm"
Oct 31 22:49:03.881374 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.879877    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-systemd-units\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.881374 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.880091    2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 22:49:03.881374 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.880125    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-run-netns\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.881374 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.880186    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-sysconfig\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.881374 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.880222    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-sys\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.881374 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.880320    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-kubernetes\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.881374 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.880400    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-host\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.881374 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.880444    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76488729-53ad-4536-8157-7d41ef49863b-system-cni-dir\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:03.881374 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.880548    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-ovnkube-config\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.881374 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.880609    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-node-log\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.881374 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.880610    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76488729-53ad-4536-8157-7d41ef49863b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:03.881374 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.880649    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/620515c1-daf9-48e9-9718-a71057b3b615-kubelet-config\") pod \"global-pull-secret-syncer-qv88p\" (UID: \"620515c1-daf9-48e9-9718-a71057b3b615\") " pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:03.881374 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.880674    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/76488729-53ad-4536-8157-7d41ef49863b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:03.881374 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.880703    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.881374 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.880679    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-log-socket\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.881374 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.880876    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-var-lib-cni-bin\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.882105 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.881308    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5f102a31-27f2-410b-9e36-2cdb8d9e852c-konnectivity-ca\") pod \"konnectivity-agent-vc2j7\" (UID: \"5f102a31-27f2-410b-9e36-2cdb8d9e852c\") " pod="kube-system/konnectivity-agent-vc2j7"
Oct 31 22:49:03.882105 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.881379    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-slash\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.882105 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.881459    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-cni-bin\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.882105 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.881669    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-env-overrides\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.882105 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.881725    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:03.882320 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.879779    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/76488729-53ad-4536-8157-7d41ef49863b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:03.882320 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.882208    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-run-ovn\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.882320 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.882247    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-var-lib-cni-multus\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.882320 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.882291    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-run-multus-certs\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.885071 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.882413    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-registration-dir\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:03.885071 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.882702    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-run-systemd\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.885071 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.882752    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-var-lib-openvswitch\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.885071 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.882771    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-system-cni-dir\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.885071 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.882828    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-modprobe-d\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.885071 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.882849    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-var-lib-kubelet\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.885071 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.882876    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-var-lib-kubelet\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.885071 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.882912    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-run-netns\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.885071 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.882949    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e46ea70b-37a4-4934-920c-ecdab212ecc6-tmp-dir\") pod \"node-resolver-cn8jx\" (UID: \"e46ea70b-37a4-4934-920c-ecdab212ecc6\") " pod="openshift-dns/node-resolver-cn8jx"
Oct 31 22:49:03.885071 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.883055    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-ovnkube-script-lib\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.885071 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.883386    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-cni-binary-copy\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.885071 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.883458    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/76488729-53ad-4536-8157-7d41ef49863b-cni-binary-copy\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:03.885071 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.883482    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-multus-daemon-config\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.885071 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:03.883540    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 22:49:03.885071 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.883550    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b1db31b-aec0-495b-8b0c-4abbde3acb28-host\") pod \"node-ca-xw8kl\" (UID: \"8b1db31b-aec0-495b-8b0c-4abbde3acb28\") " pod="openshift-image-registry/node-ca-xw8kl"
Oct 31 22:49:03.885071 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:03.883592    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/620515c1-daf9-48e9-9718-a71057b3b615-original-pull-secret podName:620515c1-daf9-48e9-9718-a71057b3b615 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:04.38357793 +0000 UTC m=+3.142024362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/620515c1-daf9-48e9-9718-a71057b3b615-original-pull-secret") pod "global-pull-secret-syncer-qv88p" (UID: "620515c1-daf9-48e9-9718-a71057b3b615") : object "kube-system"/"original-pull-secret" not registered
Oct 31 22:49:03.885071 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.883620    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-run-k8s-cni-cncf-io\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.885708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.883638    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-device-dir\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:03.885708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.883677    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-multus-conf-dir\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.885708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.883683    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-cnibin\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.885708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.883717    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/76488729-53ad-4536-8157-7d41ef49863b-cnibin\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:03.885708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.883883    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-socket-dir\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:03.885708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.884089    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8b1db31b-aec0-495b-8b0c-4abbde3acb28-serviceca\") pod \"node-ca-xw8kl\" (UID: \"8b1db31b-aec0-495b-8b0c-4abbde3acb28\") " pod="openshift-image-registry/node-ca-xw8kl"
Oct 31 22:49:03.885708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.884139    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-etc-openvswitch\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.885708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.884534    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-run-openvswitch\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.885708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.884716    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-ovn-node-metrics-cert\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.885708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.884719    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65821d7f-a625-4696-9908-44a8eb6ecdde-iptables-alerter-script\") pod \"iptables-alerter-lnhbm\" (UID: \"65821d7f-a625-4696-9908-44a8eb6ecdde\") " pod="openshift-network-operator/iptables-alerter-lnhbm"
Oct 31 22:49:03.885708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.884817    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e46ea70b-37a4-4934-920c-ecdab212ecc6-hosts-file\") pod \"node-resolver-cn8jx\" (UID: \"e46ea70b-37a4-4934-920c-ecdab212ecc6\") " pod="openshift-dns/node-resolver-cn8jx"
Oct 31 22:49:03.885708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.884856    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-kubelet\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.885708 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.884919    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5f102a31-27f2-410b-9e36-2cdb8d9e852c-agent-certs\") pod \"konnectivity-agent-vc2j7\" (UID: \"5f102a31-27f2-410b-9e36-2cdb8d9e852c\") " pod="kube-system/konnectivity-agent-vc2j7"
Oct 31 22:49:03.887503 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.887446    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/546c0de6-9661-41cf-a068-cbbf88d05c2e-tmp\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.890270 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.889674    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-tuned\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.892564 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.892543    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdr9n\" (UniqueName: \"kubernetes.io/projected/8b1db31b-aec0-495b-8b0c-4abbde3acb28-kube-api-access-hdr9n\") pod \"node-ca-xw8kl\" (UID: \"8b1db31b-aec0-495b-8b0c-4abbde3acb28\") " pod="openshift-image-registry/node-ca-xw8kl"
Oct 31 22:49:03.892641 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.892557    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bz55\" (UniqueName: \"kubernetes.io/projected/546c0de6-9661-41cf-a068-cbbf88d05c2e-kube-api-access-4bz55\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:03.894459 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.894422    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz9d9\" (UniqueName: \"kubernetes.io/projected/65821d7f-a625-4696-9908-44a8eb6ecdde-kube-api-access-mz9d9\") pod \"iptables-alerter-lnhbm\" (UID: \"65821d7f-a625-4696-9908-44a8eb6ecdde\") " pod="openshift-network-operator/iptables-alerter-lnhbm"
Oct 31 22:49:03.894459 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.894433    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spqnp\" (UniqueName: \"kubernetes.io/projected/a2ea746c-d257-4b00-be3e-ba726bd549de-kube-api-access-spqnp\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:03.896578 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.896561    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79xfm\" (UniqueName: \"kubernetes.io/projected/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-kube-api-access-79xfm\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:03.900437 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:03.900404    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 22:49:03.900437 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:03.900425    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 22:49:03.900437 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:03.900437    2575 projected.go:196] Error preparing data for projected volume kube-api-access-9hqz8 for pod openshift-network-diagnostics/network-check-target-ddxkp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 22:49:03.900606 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:03.900483    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/efec9075-8e74-475d-8160-48e356140743-kube-api-access-9hqz8 podName:efec9075-8e74-475d-8160-48e356140743 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:04.400468873 +0000 UTC m=+3.158915307 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9hqz8" (UniqueName: "kubernetes.io/projected/efec9075-8e74-475d-8160-48e356140743-kube-api-access-9hqz8") pod "network-check-target-ddxkp" (UID: "efec9075-8e74-475d-8160-48e356140743") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 22:49:03.902522 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.902496    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rwp2\" (UniqueName: \"kubernetes.io/projected/76488729-53ad-4536-8157-7d41ef49863b-kube-api-access-9rwp2\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:03.902905 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.902887    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qndh\" (UniqueName: \"kubernetes.io/projected/e46ea70b-37a4-4934-920c-ecdab212ecc6-kube-api-access-5qndh\") pod \"node-resolver-cn8jx\" (UID: \"e46ea70b-37a4-4934-920c-ecdab212ecc6\") " pod="openshift-dns/node-resolver-cn8jx"
Oct 31 22:49:03.903262 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.903233    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cxsk\" (UniqueName: \"kubernetes.io/projected/81cc6a3e-e646-4287-84a7-d6830c40784b-kube-api-access-7cxsk\") pod \"network-metrics-daemon-5qkg4\" (UID: \"81cc6a3e-e646-4287-84a7-d6830c40784b\") " pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:03.903880 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.903840    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4sx8\" (UniqueName: \"kubernetes.io/projected/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-kube-api-access-r4sx8\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:03.999815 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:03.999746    2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 22:49:04.020544 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:04.020523    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lnhbm"
Oct 31 22:49:04.026329 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:04.026302    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:04.029597 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:04.029575    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65821d7f_a625_4696_9908_44a8eb6ecdde.slice/crio-ea43db9120d0af582aeffe0e6a23e113d34310df2e662c0254186be7370b081d WatchSource:0}: Error finding container ea43db9120d0af582aeffe0e6a23e113d34310df2e662c0254186be7370b081d: Status 404 returned error can't find the container with id ea43db9120d0af582aeffe0e6a23e113d34310df2e662c0254186be7370b081d
Oct 31 22:49:04.034213 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:04.034189    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod546c0de6_9661_41cf_a068_cbbf88d05c2e.slice/crio-1c94f86140b7ea34e080bedd9f2b2d4a64281b852cce209ec89854b6a6889444 WatchSource:0}: Error finding container 1c94f86140b7ea34e080bedd9f2b2d4a64281b852cce209ec89854b6a6889444: Status 404 returned error can't find the container with id 1c94f86140b7ea34e080bedd9f2b2d4a64281b852cce209ec89854b6a6889444
Oct 31 22:49:04.034994 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:04.034974    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7pksb"
Oct 31 22:49:04.051980 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:04.051956    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:04.054058 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:04.054031    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cn8jx"
Oct 31 22:49:04.056349 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:04.056328    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vc2j7"
Oct 31 22:49:04.065616 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:04.065587    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76488729_53ad_4536_8157_7d41ef49863b.slice/crio-9ce77a3a926da5451f810783d142a98705e9ab2da818db1993298337df4d1dd1 WatchSource:0}: Error finding container 9ce77a3a926da5451f810783d142a98705e9ab2da818db1993298337df4d1dd1: Status 404 returned error can't find the container with id 9ce77a3a926da5451f810783d142a98705e9ab2da818db1993298337df4d1dd1
Oct 31 22:49:04.072514 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:04.072485    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xw8kl"
Oct 31 22:49:04.073161 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:04.073136    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f102a31_27f2_410b_9e36_2cdb8d9e852c.slice/crio-2e561ae6e12ededa8b6b6e6a3fce137075725798d98541edaa6fe26672b64795 WatchSource:0}: Error finding container 2e561ae6e12ededa8b6b6e6a3fce137075725798d98541edaa6fe26672b64795: Status 404 returned error can't find the container with id 2e561ae6e12ededa8b6b6e6a3fce137075725798d98541edaa6fe26672b64795
Oct 31 22:49:04.074032 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:04.074010    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode46ea70b_37a4_4934_920c_ecdab212ecc6.slice/crio-b78e12a84694fd87b17b1f10e7e504fd2f790910e0790be13a308a12db41ea25 WatchSource:0}: Error finding container b78e12a84694fd87b17b1f10e7e504fd2f790910e0790be13a308a12db41ea25: Status 404 returned error can't find the container with id b78e12a84694fd87b17b1f10e7e504fd2f790910e0790be13a308a12db41ea25
Oct 31 22:49:04.078457 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:04.078438    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:04.082455 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:04.082429    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b1db31b_aec0_495b_8b0c_4abbde3acb28.slice/crio-ed769345b7c93c550724a838b4f5c16a2010991acc92be55b05b2cf30a770075 WatchSource:0}: Error finding container ed769345b7c93c550724a838b4f5c16a2010991acc92be55b05b2cf30a770075: Status 404 returned error can't find the container with id ed769345b7c93c550724a838b4f5c16a2010991acc92be55b05b2cf30a770075
Oct 31 22:49:04.083584 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:04.083568    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:04.093271 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:04.093248    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0042ee_6258_4aa8_af48_7568f2f2e8a1.slice/crio-deb97a8d07b24b22d886a5dd7163e7f7136f13953737e321fb426eff2de3c50e WatchSource:0}: Error finding container deb97a8d07b24b22d886a5dd7163e7f7136f13953737e321fb426eff2de3c50e: Status 404 returned error can't find the container with id deb97a8d07b24b22d886a5dd7163e7f7136f13953737e321fb426eff2de3c50e
Oct 31 22:49:04.383166 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:04.383123    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81cc6a3e-e646-4287-84a7-d6830c40784b-metrics-certs\") pod \"network-metrics-daemon-5qkg4\" (UID: \"81cc6a3e-e646-4287-84a7-d6830c40784b\") " pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:04.383290 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:04.383262    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 22:49:04.383333 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:04.383323    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81cc6a3e-e646-4287-84a7-d6830c40784b-metrics-certs podName:81cc6a3e-e646-4287-84a7-d6830c40784b nodeName:}" failed. No retries permitted until 2026-10-31 22:49:05.383308528 +0000 UTC m=+4.141754953 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81cc6a3e-e646-4287-84a7-d6830c40784b-metrics-certs") pod "network-metrics-daemon-5qkg4" (UID: "81cc6a3e-e646-4287-84a7-d6830c40784b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 22:49:04.483647 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:04.483604    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/620515c1-daf9-48e9-9718-a71057b3b615-original-pull-secret\") pod \"global-pull-secret-syncer-qv88p\" (UID: \"620515c1-daf9-48e9-9718-a71057b3b615\") " pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:04.483801 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:04.483667    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hqz8\" (UniqueName: \"kubernetes.io/projected/efec9075-8e74-475d-8160-48e356140743-kube-api-access-9hqz8\") pod \"network-check-target-ddxkp\" (UID: \"efec9075-8e74-475d-8160-48e356140743\") " pod="openshift-network-diagnostics/network-check-target-ddxkp"
Oct 31 22:49:04.483801 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:04.483795    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 22:49:04.483902 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:04.483816    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 22:49:04.483902 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:04.483830    2575 projected.go:196] Error preparing data for projected volume kube-api-access-9hqz8 for pod openshift-network-diagnostics/network-check-target-ddxkp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 22:49:04.483902 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:04.483882    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/efec9075-8e74-475d-8160-48e356140743-kube-api-access-9hqz8 podName:efec9075-8e74-475d-8160-48e356140743 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:05.48386694 +0000 UTC m=+4.242313374 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9hqz8" (UniqueName: "kubernetes.io/projected/efec9075-8e74-475d-8160-48e356140743-kube-api-access-9hqz8") pod "network-check-target-ddxkp" (UID: "efec9075-8e74-475d-8160-48e356140743") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 22:49:04.484231 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:04.484217    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 22:49:04.484285 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:04.484258    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/620515c1-daf9-48e9-9718-a71057b3b615-original-pull-secret podName:620515c1-daf9-48e9-9718-a71057b3b615 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:05.484245333 +0000 UTC m=+4.242691767 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/620515c1-daf9-48e9-9718-a71057b3b615-original-pull-secret") pod "global-pull-secret-syncer-qv88p" (UID: "620515c1-daf9-48e9-9718-a71057b3b615") : object "kube-system"/"original-pull-secret" not registered
Oct 31 22:49:04.726310 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:04.726201    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 22:44:03 +0000 UTC" deadline="2028-06-15 15:52:18.630494759 +0000 UTC"
Oct 31 22:49:04.726310 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:04.726308    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14225h3m13.904191396s"
Oct 31 22:49:04.753663 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:04.753624    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-163.us-east-2.compute.internal" event={"ID":"2a808dc7f3e00ce75e0f6c4e3f550fac","Type":"ContainerStarted","Data":"8a9b910b0f109fa203a299360f5e977a4ff3c231ffe2f1f4f339f15024163c9f"}
Oct 31 22:49:04.757853 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:04.757810    2575 generic.go:356] "Generic (PLEG): container finished" podID="8ec0c93a8e98312f4e27fee411631fef" containerID="a2ea395c9a9dd176750af59537229c9d91eff30fd976a82fd9ec502edc005c42" exitCode=0
Oct 31 22:49:04.757979 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:04.757895    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal" event={"ID":"8ec0c93a8e98312f4e27fee411631fef","Type":"ContainerDied","Data":"a2ea395c9a9dd176750af59537229c9d91eff30fd976a82fd9ec502edc005c42"}
Oct 31 22:49:04.766067 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:04.766016    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xw8kl" event={"ID":"8b1db31b-aec0-495b-8b0c-4abbde3acb28","Type":"ContainerStarted","Data":"ed769345b7c93c550724a838b4f5c16a2010991acc92be55b05b2cf30a770075"}
Oct 31 22:49:04.775282 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:04.775220    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7pksb" event={"ID":"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16","Type":"ContainerStarted","Data":"4150b4496c8eefc7e46cde1c0ecf2e8357f6f408e516de83d24896e615ad921d"}
Oct 31 22:49:04.781293 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:04.781239    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dcczv" event={"ID":"546c0de6-9661-41cf-a068-cbbf88d05c2e","Type":"ContainerStarted","Data":"1c94f86140b7ea34e080bedd9f2b2d4a64281b852cce209ec89854b6a6889444"}
Oct 31 22:49:04.787621 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:04.787532    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2" event={"ID":"a2ea746c-d257-4b00-be3e-ba726bd549de","Type":"ContainerStarted","Data":"a084ce34ca6837ff5bc61c82598ceaa97d2b8a06e6f7f12be6d8c19674a7bd34"}
Oct 31 22:49:04.789548 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:04.788345    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-163.us-east-2.compute.internal" podStartSLOduration=2.788331795 podStartE2EDuration="2.788331795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:04.771226714 +0000 UTC m=+3.529673156" watchObservedRunningTime="2026-10-31 22:49:04.788331795 +0000 UTC m=+3.546778236"
Oct 31 22:49:04.800034 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:04.799774    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl" event={"ID":"1f0042ee-6258-4aa8-af48-7568f2f2e8a1","Type":"ContainerStarted","Data":"deb97a8d07b24b22d886a5dd7163e7f7136f13953737e321fb426eff2de3c50e"}
Oct 31 22:49:04.803595 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:04.803570    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cn8jx" event={"ID":"e46ea70b-37a4-4934-920c-ecdab212ecc6","Type":"ContainerStarted","Data":"b78e12a84694fd87b17b1f10e7e504fd2f790910e0790be13a308a12db41ea25"}
Oct 31 22:49:04.814527 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:04.814503    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vc2j7" event={"ID":"5f102a31-27f2-410b-9e36-2cdb8d9e852c","Type":"ContainerStarted","Data":"2e561ae6e12ededa8b6b6e6a3fce137075725798d98541edaa6fe26672b64795"}
Oct 31 22:49:04.829657 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:04.829621    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2s8p8" event={"ID":"76488729-53ad-4536-8157-7d41ef49863b","Type":"ContainerStarted","Data":"9ce77a3a926da5451f810783d142a98705e9ab2da818db1993298337df4d1dd1"}
Oct 31 22:49:04.845216 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:04.845172    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lnhbm" event={"ID":"65821d7f-a625-4696-9908-44a8eb6ecdde","Type":"ContainerStarted","Data":"ea43db9120d0af582aeffe0e6a23e113d34310df2e662c0254186be7370b081d"}
Oct 31 22:49:05.391102 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:05.391036    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81cc6a3e-e646-4287-84a7-d6830c40784b-metrics-certs\") pod \"network-metrics-daemon-5qkg4\" (UID: \"81cc6a3e-e646-4287-84a7-d6830c40784b\") " pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:05.391327 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:05.391256    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 22:49:05.391327 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:05.391321    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81cc6a3e-e646-4287-84a7-d6830c40784b-metrics-certs podName:81cc6a3e-e646-4287-84a7-d6830c40784b nodeName:}" failed. No retries permitted until 2026-10-31 22:49:07.391298485 +0000 UTC m=+6.149744910 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81cc6a3e-e646-4287-84a7-d6830c40784b-metrics-certs") pod "network-metrics-daemon-5qkg4" (UID: "81cc6a3e-e646-4287-84a7-d6830c40784b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 22:49:05.491672 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:05.491620    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/620515c1-daf9-48e9-9718-a71057b3b615-original-pull-secret\") pod \"global-pull-secret-syncer-qv88p\" (UID: \"620515c1-daf9-48e9-9718-a71057b3b615\") " pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:05.491833 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:05.491684    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hqz8\" (UniqueName: \"kubernetes.io/projected/efec9075-8e74-475d-8160-48e356140743-kube-api-access-9hqz8\") pod \"network-check-target-ddxkp\" (UID: \"efec9075-8e74-475d-8160-48e356140743\") " pod="openshift-network-diagnostics/network-check-target-ddxkp"
Oct 31 22:49:05.491902 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:05.491883    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 22:49:05.491950 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:05.491912    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 22:49:05.491950 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:05.491925    2575 projected.go:196] Error preparing data for projected volume kube-api-access-9hqz8 for pod openshift-network-diagnostics/network-check-target-ddxkp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 22:49:05.492040 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:05.491977    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/efec9075-8e74-475d-8160-48e356140743-kube-api-access-9hqz8 podName:efec9075-8e74-475d-8160-48e356140743 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:07.491960675 +0000 UTC m=+6.250407110 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9hqz8" (UniqueName: "kubernetes.io/projected/efec9075-8e74-475d-8160-48e356140743-kube-api-access-9hqz8") pod "network-check-target-ddxkp" (UID: "efec9075-8e74-475d-8160-48e356140743") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 22:49:05.492374 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:05.492341    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 22:49:05.492456 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:05.492435    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/620515c1-daf9-48e9-9718-a71057b3b615-original-pull-secret podName:620515c1-daf9-48e9-9718-a71057b3b615 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:07.492421068 +0000 UTC m=+6.250867490 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/620515c1-daf9-48e9-9718-a71057b3b615-original-pull-secret") pod "global-pull-secret-syncer-qv88p" (UID: "620515c1-daf9-48e9-9718-a71057b3b615") : object "kube-system"/"original-pull-secret" not registered
Oct 31 22:49:05.726950 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:05.726866    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 22:44:03 +0000 UTC" deadline="2028-07-03 23:39:16.649223184 +0000 UTC"
Oct 31 22:49:05.726950 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:05.726909    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14664h50m10.922318762s"
Oct 31 22:49:05.740557 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:05.740530    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:05.740708 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:05.740653    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv88p" podUID="620515c1-daf9-48e9-9718-a71057b3b615"
Oct 31 22:49:05.741039 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:05.741021    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddxkp"
Oct 31 22:49:05.741115 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:05.741103    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddxkp" podUID="efec9075-8e74-475d-8160-48e356140743"
Oct 31 22:49:05.741201 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:05.741172    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:05.741281 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:05.741256    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qkg4" podUID="81cc6a3e-e646-4287-84a7-d6830c40784b"
Oct 31 22:49:05.862476 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:05.862440    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal" event={"ID":"8ec0c93a8e98312f4e27fee411631fef","Type":"ContainerStarted","Data":"ef7c36772239aa412e173bfb2f7b2567d71bb227a4367b1dbfa25382fc5fe93c"}
Oct 31 22:49:05.880149 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:05.880098    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal" podStartSLOduration=3.880082758 podStartE2EDuration="3.880082758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:05.879736436 +0000 UTC m=+4.638182876" watchObservedRunningTime="2026-10-31 22:49:05.880082758 +0000 UTC m=+4.638529200"
Oct 31 22:49:06.867115 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:06.866870    2575 generic.go:356] "Generic (PLEG): container finished" podID="76488729-53ad-4536-8157-7d41ef49863b" containerID="a5b8955efe0b96a7dd03bdb9a68acaec6c0734e0ce4f50fe2cef3fe92748cfce" exitCode=0
Oct 31 22:49:06.867850 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:06.867823    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2s8p8" event={"ID":"76488729-53ad-4536-8157-7d41ef49863b","Type":"ContainerDied","Data":"a5b8955efe0b96a7dd03bdb9a68acaec6c0734e0ce4f50fe2cef3fe92748cfce"}
Oct 31 22:49:07.409036 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:07.409001    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81cc6a3e-e646-4287-84a7-d6830c40784b-metrics-certs\") pod \"network-metrics-daemon-5qkg4\" (UID: \"81cc6a3e-e646-4287-84a7-d6830c40784b\") " pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:07.409235 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:07.409217    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 22:49:07.409316 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:07.409303    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81cc6a3e-e646-4287-84a7-d6830c40784b-metrics-certs podName:81cc6a3e-e646-4287-84a7-d6830c40784b nodeName:}" failed. No retries permitted until 2026-10-31 22:49:11.409266821 +0000 UTC m=+10.167713260 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81cc6a3e-e646-4287-84a7-d6830c40784b-metrics-certs") pod "network-metrics-daemon-5qkg4" (UID: "81cc6a3e-e646-4287-84a7-d6830c40784b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 22:49:07.510327 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:07.510277    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hqz8\" (UniqueName: \"kubernetes.io/projected/efec9075-8e74-475d-8160-48e356140743-kube-api-access-9hqz8\") pod \"network-check-target-ddxkp\" (UID: \"efec9075-8e74-475d-8160-48e356140743\") " pod="openshift-network-diagnostics/network-check-target-ddxkp"
Oct 31 22:49:07.510562 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:07.510386    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/620515c1-daf9-48e9-9718-a71057b3b615-original-pull-secret\") pod \"global-pull-secret-syncer-qv88p\" (UID: \"620515c1-daf9-48e9-9718-a71057b3b615\") " pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:07.510562 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:07.510506    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 22:49:07.510656 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:07.510569    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/620515c1-daf9-48e9-9718-a71057b3b615-original-pull-secret podName:620515c1-daf9-48e9-9718-a71057b3b615 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:11.510552583 +0000 UTC m=+10.268999023 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/620515c1-daf9-48e9-9718-a71057b3b615-original-pull-secret") pod "global-pull-secret-syncer-qv88p" (UID: "620515c1-daf9-48e9-9718-a71057b3b615") : object "kube-system"/"original-pull-secret" not registered
Oct 31 22:49:07.510965 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:07.510945    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 22:49:07.510965 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:07.510966    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 22:49:07.511103 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:07.510975    2575 projected.go:196] Error preparing data for projected volume kube-api-access-9hqz8 for pod openshift-network-diagnostics/network-check-target-ddxkp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 22:49:07.511103 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:07.511007    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/efec9075-8e74-475d-8160-48e356140743-kube-api-access-9hqz8 podName:efec9075-8e74-475d-8160-48e356140743 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:11.510995442 +0000 UTC m=+10.269441874 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9hqz8" (UniqueName: "kubernetes.io/projected/efec9075-8e74-475d-8160-48e356140743-kube-api-access-9hqz8") pod "network-check-target-ddxkp" (UID: "efec9075-8e74-475d-8160-48e356140743") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 22:49:07.739594 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:07.739524    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:07.739728 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:07.739661    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qkg4" podUID="81cc6a3e-e646-4287-84a7-d6830c40784b"
Oct 31 22:49:07.739977 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:07.739955    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:07.740080 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:07.740055    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv88p" podUID="620515c1-daf9-48e9-9718-a71057b3b615"
Oct 31 22:49:07.740132 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:07.740109    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddxkp"
Oct 31 22:49:07.740183 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:07.740170    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddxkp" podUID="efec9075-8e74-475d-8160-48e356140743"
Oct 31 22:49:07.871419 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:07.871385    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vc2j7" event={"ID":"5f102a31-27f2-410b-9e36-2cdb8d9e852c","Type":"ContainerStarted","Data":"2254487342ed81b93b5a39a4dc7a7161d9e25242ef886092afc9a740aaa096db"}
Oct 31 22:49:08.884101 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:08.884032    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xw8kl" event={"ID":"8b1db31b-aec0-495b-8b0c-4abbde3acb28","Type":"ContainerStarted","Data":"c821e7d29b0d0f5bd3afe3bf1aa95e068c3b5d687a78aa56b4a1d59e23e7ea51"}
Oct 31 22:49:08.903154 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:08.902316    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vc2j7" podStartSLOduration=4.371271958 podStartE2EDuration="7.902301212s" totalImagesPullingTime="3.531029254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:49:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:07.892243642 +0000 UTC m=+6.650690084" watchObservedRunningTime="2026-10-31 22:49:08.902301212 +0000 UTC m=+7.660747653"
Oct 31 22:49:09.676132 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:09.675955    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vc2j7"
Oct 31 22:49:09.676779 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:09.676549    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vc2j7"
Oct 31 22:49:09.696094 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:09.695716    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xw8kl" podStartSLOduration=4.400805189 podStartE2EDuration="8.695697926s" totalImagesPullingTime="4.294892737s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:49:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:08.903119377 +0000 UTC m=+7.661565826" watchObservedRunningTime="2026-10-31 22:49:09.695697926 +0000 UTC m=+8.454144371"
Oct 31 22:49:09.741187 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:09.740398    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddxkp"
Oct 31 22:49:09.741187 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:09.740534    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddxkp" podUID="efec9075-8e74-475d-8160-48e356140743"
Oct 31 22:49:09.741187 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:09.740910    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:09.741187 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:09.741009    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qkg4" podUID="81cc6a3e-e646-4287-84a7-d6830c40784b"
Oct 31 22:49:09.741187 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:09.741065    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:09.741187 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:09.741135    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv88p" podUID="620515c1-daf9-48e9-9718-a71057b3b615"
Oct 31 22:49:09.888911 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:09.888447    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2" event={"ID":"a2ea746c-d257-4b00-be3e-ba726bd549de","Type":"ContainerStarted","Data":"10553d7f2c691259ebf221d21b1ed3b929ba71284d9864ce005109c09661a3db"}
Oct 31 22:49:10.890899 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:10.890316    2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 22:49:10.976206 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:10.976187    2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 22:49:11.445620 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:11.445570    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81cc6a3e-e646-4287-84a7-d6830c40784b-metrics-certs\") pod \"network-metrics-daemon-5qkg4\" (UID: \"81cc6a3e-e646-4287-84a7-d6830c40784b\") " pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:11.445776 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:11.445735    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 22:49:11.445826 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:11.445799    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81cc6a3e-e646-4287-84a7-d6830c40784b-metrics-certs podName:81cc6a3e-e646-4287-84a7-d6830c40784b nodeName:}" failed. No retries permitted until 2026-10-31 22:49:19.445781934 +0000 UTC m=+18.204228369 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81cc6a3e-e646-4287-84a7-d6830c40784b-metrics-certs") pod "network-metrics-daemon-5qkg4" (UID: "81cc6a3e-e646-4287-84a7-d6830c40784b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 22:49:11.548952 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:11.548123    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/620515c1-daf9-48e9-9718-a71057b3b615-original-pull-secret\") pod \"global-pull-secret-syncer-qv88p\" (UID: \"620515c1-daf9-48e9-9718-a71057b3b615\") " pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:11.548952 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:11.548184    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hqz8\" (UniqueName: \"kubernetes.io/projected/efec9075-8e74-475d-8160-48e356140743-kube-api-access-9hqz8\") pod \"network-check-target-ddxkp\" (UID: \"efec9075-8e74-475d-8160-48e356140743\") " pod="openshift-network-diagnostics/network-check-target-ddxkp"
Oct 31 22:49:11.548952 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:11.548348    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 22:49:11.548952 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:11.548402    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 22:49:11.548952 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:11.548415    2575 projected.go:196] Error preparing data for projected volume kube-api-access-9hqz8 for pod openshift-network-diagnostics/network-check-target-ddxkp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 22:49:11.548952 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:11.548476    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/efec9075-8e74-475d-8160-48e356140743-kube-api-access-9hqz8 podName:efec9075-8e74-475d-8160-48e356140743 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:19.548458132 +0000 UTC m=+18.306904569 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9hqz8" (UniqueName: "kubernetes.io/projected/efec9075-8e74-475d-8160-48e356140743-kube-api-access-9hqz8") pod "network-check-target-ddxkp" (UID: "efec9075-8e74-475d-8160-48e356140743") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 22:49:11.548952 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:11.548865    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 22:49:11.548952 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:11.548916    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/620515c1-daf9-48e9-9718-a71057b3b615-original-pull-secret podName:620515c1-daf9-48e9-9718-a71057b3b615 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:19.548901977 +0000 UTC m=+18.307348418 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/620515c1-daf9-48e9-9718-a71057b3b615-original-pull-secret") pod "global-pull-secret-syncer-qv88p" (UID: "620515c1-daf9-48e9-9718-a71057b3b615") : object "kube-system"/"original-pull-secret" not registered
Oct 31 22:49:11.740957 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:11.740930    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:11.741084 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:11.741026    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:11.741084 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:11.741061    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv88p" podUID="620515c1-daf9-48e9-9718-a71057b3b615"
Oct 31 22:49:11.741167 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:11.741101    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qkg4" podUID="81cc6a3e-e646-4287-84a7-d6830c40784b"
Oct 31 22:49:11.741167 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:11.741125    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddxkp"
Oct 31 22:49:11.741241 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:11.741175    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddxkp" podUID="efec9075-8e74-475d-8160-48e356140743"
Oct 31 22:49:11.747185 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:11.747076    2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T22:49:10.976202454Z","UUID":"d77baec5-7912-4378-b07d-17263daf57dd","Handler":null,"Name":"","Endpoint":""}
Oct 31 22:49:11.749916 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:11.749893    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 22:49:11.750028 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:11.749955    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 22:49:11.897051 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:11.897009    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7pksb" event={"ID":"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16","Type":"ContainerStarted","Data":"7b27b2233c3773e597a65e2f1b3d18679952173e15dc2336cf9e7b6d93761a69"}
Oct 31 22:49:11.903560 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:11.903528    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2" event={"ID":"a2ea746c-d257-4b00-be3e-ba726bd549de","Type":"ContainerStarted","Data":"ee43a45990c9a51274442543bb83922585705ad955d294a14aa2093cbc39aa74"}
Oct 31 22:49:11.905826 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:11.905766    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cn8jx" event={"ID":"e46ea70b-37a4-4934-920c-ecdab212ecc6","Type":"ContainerStarted","Data":"199b30af128483792427acc6970c7d1037d764aaf764f960915aedc24a5728ed"}
Oct 31 22:49:11.909778 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:11.909747    2575 generic.go:356] "Generic (PLEG): container finished" podID="76488729-53ad-4536-8157-7d41ef49863b" containerID="e057954cc32c812adb180af8da989c99bce5340f07ac0bd22d1a21a8a25c3de7" exitCode=0
Oct 31 22:49:11.909901 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:11.909787    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2s8p8" event={"ID":"76488729-53ad-4536-8157-7d41ef49863b","Type":"ContainerDied","Data":"e057954cc32c812adb180af8da989c99bce5340f07ac0bd22d1a21a8a25c3de7"}
Oct 31 22:49:11.931628 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:11.930595    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-7pksb" podStartSLOduration=3.732279817 podStartE2EDuration="10.930577096s" totalImagesPullingTime="7.198297279s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:49:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:11.928952919 +0000 UTC m=+10.687399364" watchObservedRunningTime="2026-10-31 22:49:11.930577096 +0000 UTC m=+10.689023541"
Oct 31 22:49:12.915964 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:12.915721    2575 generic.go:356] "Generic (PLEG): container finished" podID="76488729-53ad-4536-8157-7d41ef49863b" containerID="1219019f7b6709dc5fdcfd50b093de9a96fdb6b48ad567f89c3fe0ef4953acd6" exitCode=0
Oct 31 22:49:12.916472 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:12.915817    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2s8p8" event={"ID":"76488729-53ad-4536-8157-7d41ef49863b","Type":"ContainerDied","Data":"1219019f7b6709dc5fdcfd50b093de9a96fdb6b48ad567f89c3fe0ef4953acd6"}
Oct 31 22:49:12.919661 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:12.919632    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2" event={"ID":"a2ea746c-d257-4b00-be3e-ba726bd549de","Type":"ContainerStarted","Data":"3b759525a770366c6fd2b2e9429cc24d5464787883aea0c21951e29a7f1821b0"}
Oct 31 22:49:12.961308 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:12.961259    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-cn8jx" podStartSLOduration=4.538816291 podStartE2EDuration="11.96123916s" totalImagesPullingTime="7.422422869s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:49:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:11.989756348 +0000 UTC m=+10.748202794" watchObservedRunningTime="2026-10-31 22:49:12.96123916 +0000 UTC m=+11.719685606"
Oct 31 22:49:12.986133 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:12.986080    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2" podStartSLOduration=4.361597836 podStartE2EDuration="11.986067191s" totalImagesPullingTime="7.624469355s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:49:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:12.984997041 +0000 UTC m=+11.743443477" watchObservedRunningTime="2026-10-31 22:49:12.986067191 +0000 UTC m=+11.744513633"
Oct 31 22:49:13.738689 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:13.738650    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:13.738689 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:13.738674    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddxkp"
Oct 31 22:49:13.738908 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:13.738775    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qkg4" podUID="81cc6a3e-e646-4287-84a7-d6830c40784b"
Oct 31 22:49:13.738908 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:13.738842    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:13.739016 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:13.738941    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv88p" podUID="620515c1-daf9-48e9-9718-a71057b3b615"
Oct 31 22:49:13.739069 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:13.739023    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddxkp" podUID="efec9075-8e74-475d-8160-48e356140743"
Oct 31 22:49:13.924049 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:13.924012    2575 generic.go:356] "Generic (PLEG): container finished" podID="76488729-53ad-4536-8157-7d41ef49863b" containerID="5c8dbb094f04552530ef9c5a3e9b96095b84b36434931c518345fde7d662a65a" exitCode=0
Oct 31 22:49:13.924783 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:13.924107    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2s8p8" event={"ID":"76488729-53ad-4536-8157-7d41ef49863b","Type":"ContainerDied","Data":"5c8dbb094f04552530ef9c5a3e9b96095b84b36434931c518345fde7d662a65a"}
Oct 31 22:49:13.926258 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:13.926232    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lnhbm" event={"ID":"65821d7f-a625-4696-9908-44a8eb6ecdde","Type":"ContainerStarted","Data":"9d9b54b6bf28cf5cb0578711692358b2cf4f7eaa98200a9a934a749ab4474ef3"}
Oct 31 22:49:13.927794 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:13.927769    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dcczv" event={"ID":"546c0de6-9661-41cf-a068-cbbf88d05c2e","Type":"ContainerStarted","Data":"c7372bf365f41bfac6b9ad01ffda754c42b931a664fd8017ed9c0cdad0b9944e"}
Oct 31 22:49:13.972044 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:13.971991    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lnhbm" podStartSLOduration=5.407310827 podStartE2EDuration="12.971975047s" totalImagesPullingTime="7.56466422s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:49:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:13.9715108 +0000 UTC m=+12.729957255" watchObservedRunningTime="2026-10-31 22:49:13.971975047 +0000 UTC m=+12.730421491"
Oct 31 22:49:13.993373 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:13.993310    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dcczv" podStartSLOduration=3.7255640789999998 podStartE2EDuration="12.993297153s" totalImagesPullingTime="9.267733074s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:49:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:13.99255579 +0000 UTC m=+12.751002237" watchObservedRunningTime="2026-10-31 22:49:13.993297153 +0000 UTC m=+12.751743597"
Oct 31 22:49:15.738012 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:15.737983    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:15.738652 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:15.737992    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:15.738652 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:15.738114    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qkg4" podUID="81cc6a3e-e646-4287-84a7-d6830c40784b"
Oct 31 22:49:15.738652 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:15.738166    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv88p" podUID="620515c1-daf9-48e9-9718-a71057b3b615"
Oct 31 22:49:15.738652 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:15.738187    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddxkp"
Oct 31 22:49:15.738652 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:15.738261    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddxkp" podUID="efec9075-8e74-475d-8160-48e356140743"
Oct 31 22:49:16.937817 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:16.937574    2575 generic.go:356] "Generic (PLEG): container finished" podID="76488729-53ad-4536-8157-7d41ef49863b" containerID="cdb3c230eaf4f87f9ba4cdcd9fb312d0c5c39dfa6f791c199f5c56ed4036357b" exitCode=0
Oct 31 22:49:16.938250 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:16.937632    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2s8p8" event={"ID":"76488729-53ad-4536-8157-7d41ef49863b","Type":"ContainerDied","Data":"cdb3c230eaf4f87f9ba4cdcd9fb312d0c5c39dfa6f791c199f5c56ed4036357b"}
Oct 31 22:49:17.741069 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:17.741037    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddxkp"
Oct 31 22:49:17.741069 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:17.741055    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:17.741069 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:17.741048    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:17.741287 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:17.741153    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qkg4" podUID="81cc6a3e-e646-4287-84a7-d6830c40784b"
Oct 31 22:49:17.741287 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:17.741207    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv88p" podUID="620515c1-daf9-48e9-9718-a71057b3b615"
Oct 31 22:49:17.741395 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:17.741291    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddxkp" podUID="efec9075-8e74-475d-8160-48e356140743"
Oct 31 22:49:17.942231 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:17.942194    2575 generic.go:356] "Generic (PLEG): container finished" podID="76488729-53ad-4536-8157-7d41ef49863b" containerID="9fcecf831bfc119e14455eb7a55efb32a3f759f5891de634ffe6b3a13b761256" exitCode=0
Oct 31 22:49:17.942669 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:17.942320    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2s8p8" event={"ID":"76488729-53ad-4536-8157-7d41ef49863b","Type":"ContainerDied","Data":"9fcecf831bfc119e14455eb7a55efb32a3f759f5891de634ffe6b3a13b761256"}
Oct 31 22:49:17.944770 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:17.944747    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl" event={"ID":"1f0042ee-6258-4aa8-af48-7568f2f2e8a1","Type":"ContainerStarted","Data":"a192bd72d8082efb5034f801ff5a814b6a35d698cff6318cc13c92fd3e132d07"}
Oct 31 22:49:17.944864 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:17.944820    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl" event={"ID":"1f0042ee-6258-4aa8-af48-7568f2f2e8a1","Type":"ContainerStarted","Data":"775e18e7d19c382a772703b80670abe8a5ee8a1adc8d381263d0f1d7c4bf16c5"}
Oct 31 22:49:18.950349 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:18.950131    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2s8p8" event={"ID":"76488729-53ad-4536-8157-7d41ef49863b","Type":"ContainerStarted","Data":"6668b4a9d975e283da2033fcdc27ad3803d0611eb86da33f6719ac5f472e8b94"}
Oct 31 22:49:18.952673 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:18.952654    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s9nl_1f0042ee-6258-4aa8-af48-7568f2f2e8a1/ovn-acl-logging/0.log"
Oct 31 22:49:18.952979 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:18.952960    2575 generic.go:356] "Generic (PLEG): container finished" podID="1f0042ee-6258-4aa8-af48-7568f2f2e8a1" containerID="a192bd72d8082efb5034f801ff5a814b6a35d698cff6318cc13c92fd3e132d07" exitCode=1
Oct 31 22:49:18.953024 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:18.952989    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl" event={"ID":"1f0042ee-6258-4aa8-af48-7568f2f2e8a1","Type":"ContainerDied","Data":"a192bd72d8082efb5034f801ff5a814b6a35d698cff6318cc13c92fd3e132d07"}
Oct 31 22:49:18.953057 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:18.953034    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl" event={"ID":"1f0042ee-6258-4aa8-af48-7568f2f2e8a1","Type":"ContainerStarted","Data":"ce97db22df1a1d13125cfd03148303d2868c7dea199845c8e9d039239c2fda05"}
Oct 31 22:49:18.953088 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:18.953065    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl" event={"ID":"1f0042ee-6258-4aa8-af48-7568f2f2e8a1","Type":"ContainerStarted","Data":"99f38e13c4cee9ab2edbe92b75f352f62827e3dc4e2805afe87ee1beaf22e02a"}
Oct 31 22:49:18.953119 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:18.953092    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl" event={"ID":"1f0042ee-6258-4aa8-af48-7568f2f2e8a1","Type":"ContainerStarted","Data":"1a1e2d3b14564fcd3495e313fdfc65f6bcb9bcc21893f569d666c3f0879603f4"}
Oct 31 22:49:18.953152 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:18.953119    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl" event={"ID":"1f0042ee-6258-4aa8-af48-7568f2f2e8a1","Type":"ContainerStarted","Data":"13e1acb87b8d4243de9d9bafe5d98ddd2beee488ea7ccfe0d9391aa45bda48b1"}
Oct 31 22:49:18.981551 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:18.981509    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2s8p8" podStartSLOduration=7.876515255 podStartE2EDuration="17.981497182s" totalImagesPullingTime="10.104981927s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:49:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:18.981049836 +0000 UTC m=+17.739496281" watchObservedRunningTime="2026-10-31 22:49:18.981497182 +0000 UTC m=+17.739943626"
Oct 31 22:49:19.511806 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:19.511773    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81cc6a3e-e646-4287-84a7-d6830c40784b-metrics-certs\") pod \"network-metrics-daemon-5qkg4\" (UID: \"81cc6a3e-e646-4287-84a7-d6830c40784b\") " pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:19.511972 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:19.511906    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 22:49:19.511972 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:19.511953    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81cc6a3e-e646-4287-84a7-d6830c40784b-metrics-certs podName:81cc6a3e-e646-4287-84a7-d6830c40784b nodeName:}" failed. No retries permitted until 2026-10-31 22:49:35.511941592 +0000 UTC m=+34.270388013 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81cc6a3e-e646-4287-84a7-d6830c40784b-metrics-certs") pod "network-metrics-daemon-5qkg4" (UID: "81cc6a3e-e646-4287-84a7-d6830c40784b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 22:49:19.612459 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:19.612431    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hqz8\" (UniqueName: \"kubernetes.io/projected/efec9075-8e74-475d-8160-48e356140743-kube-api-access-9hqz8\") pod \"network-check-target-ddxkp\" (UID: \"efec9075-8e74-475d-8160-48e356140743\") " pod="openshift-network-diagnostics/network-check-target-ddxkp"
Oct 31 22:49:19.612606 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:19.612484    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/620515c1-daf9-48e9-9718-a71057b3b615-original-pull-secret\") pod \"global-pull-secret-syncer-qv88p\" (UID: \"620515c1-daf9-48e9-9718-a71057b3b615\") " pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:19.612606 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:19.612569    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 22:49:19.612606 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:19.612599    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 22:49:19.612752 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:19.612614    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/620515c1-daf9-48e9-9718-a71057b3b615-original-pull-secret podName:620515c1-daf9-48e9-9718-a71057b3b615 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:35.61260186 +0000 UTC m=+34.371048296 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/620515c1-daf9-48e9-9718-a71057b3b615-original-pull-secret") pod "global-pull-secret-syncer-qv88p" (UID: "620515c1-daf9-48e9-9718-a71057b3b615") : object "kube-system"/"original-pull-secret" not registered
Oct 31 22:49:19.612752 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:19.612620    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 22:49:19.612752 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:19.612635    2575 projected.go:196] Error preparing data for projected volume kube-api-access-9hqz8 for pod openshift-network-diagnostics/network-check-target-ddxkp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 22:49:19.612752 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:19.612685    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/efec9075-8e74-475d-8160-48e356140743-kube-api-access-9hqz8 podName:efec9075-8e74-475d-8160-48e356140743 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:35.612669618 +0000 UTC m=+34.371116059 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9hqz8" (UniqueName: "kubernetes.io/projected/efec9075-8e74-475d-8160-48e356140743-kube-api-access-9hqz8") pod "network-check-target-ddxkp" (UID: "efec9075-8e74-475d-8160-48e356140743") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 22:49:19.737564 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:19.737530    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:19.737564 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:19.737552    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddxkp"
Oct 31 22:49:19.737738 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:19.737600    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:19.737738 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:19.737675    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddxkp" podUID="efec9075-8e74-475d-8160-48e356140743"
Oct 31 22:49:19.737844 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:19.737822    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qkg4" podUID="81cc6a3e-e646-4287-84a7-d6830c40784b"
Oct 31 22:49:19.737929 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:19.737911    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv88p" podUID="620515c1-daf9-48e9-9718-a71057b3b615"
Oct 31 22:49:20.962203 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:20.962175    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s9nl_1f0042ee-6258-4aa8-af48-7568f2f2e8a1/ovn-acl-logging/0.log"
Oct 31 22:49:20.962719 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:20.962553    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl" event={"ID":"1f0042ee-6258-4aa8-af48-7568f2f2e8a1","Type":"ContainerStarted","Data":"94bd1e8fa577a135ece8a75c6166a30d897b0170cdf478eaa3858fd10691657b"}
Oct 31 22:49:21.742512 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:21.742478    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddxkp"
Oct 31 22:49:21.742688 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:21.742550    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:21.742688 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:21.742587    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddxkp" podUID="efec9075-8e74-475d-8160-48e356140743"
Oct 31 22:49:21.742688 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:21.742616    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qkg4" podUID="81cc6a3e-e646-4287-84a7-d6830c40784b"
Oct 31 22:49:21.742688 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:21.742649    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:21.742842 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:21.742710    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv88p" podUID="620515c1-daf9-48e9-9718-a71057b3b615"
Oct 31 22:49:22.133087 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:22.133062    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vc2j7"
Oct 31 22:49:22.133548 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:22.133164    2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 22:49:22.133669 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:22.133648    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vc2j7"
Oct 31 22:49:22.969833 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:22.969653    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s9nl_1f0042ee-6258-4aa8-af48-7568f2f2e8a1/ovn-acl-logging/0.log"
Oct 31 22:49:22.970190 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:22.970167    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl" event={"ID":"1f0042ee-6258-4aa8-af48-7568f2f2e8a1","Type":"ContainerStarted","Data":"d7f8f5deba217a61291afe237d17e7e061f5c21735bec7f02cbc086162200f30"}
Oct 31 22:49:22.970505 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:22.970478    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:22.970699 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:22.970548    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:22.970699 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:22.970570    2575 scope.go:122] "RemoveContainer" containerID="a192bd72d8082efb5034f801ff5a814b6a35d698cff6318cc13c92fd3e132d07"
Oct 31 22:49:22.970699 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:22.970594    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:22.987327 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:22.987303    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:22.987484 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:22.987472    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:23.737658 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:23.737624    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:23.738255 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:23.737625    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddxkp"
Oct 31 22:49:23.738255 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:23.737737    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:23.738255 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:23.737733    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qkg4" podUID="81cc6a3e-e646-4287-84a7-d6830c40784b"
Oct 31 22:49:23.738255 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:23.737825    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddxkp" podUID="efec9075-8e74-475d-8160-48e356140743"
Oct 31 22:49:23.738255 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:23.737898    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv88p" podUID="620515c1-daf9-48e9-9718-a71057b3b615"
Oct 31 22:49:23.975968 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:23.975944    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s9nl_1f0042ee-6258-4aa8-af48-7568f2f2e8a1/ovn-acl-logging/0.log"
Oct 31 22:49:23.976338 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:23.976308    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl" event={"ID":"1f0042ee-6258-4aa8-af48-7568f2f2e8a1","Type":"ContainerStarted","Data":"715b6d780f7321a8e633501292bcc98529d2c5157390c6bab1c8ebddff28a276"}
Oct 31 22:49:24.016519 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:24.016433    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl" podStartSLOduration=9.361998592 podStartE2EDuration="23.016418776s" totalImagesPullingTime="13.654420184s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:49:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:24.01616659 +0000 UTC m=+22.774613033" watchObservedRunningTime="2026-10-31 22:49:24.016418776 +0000 UTC m=+22.774865220"
Oct 31 22:49:24.208309 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:24.208278    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qv88p"]
Oct 31 22:49:24.208499 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:24.208446    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:24.208558 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:24.208540    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv88p" podUID="620515c1-daf9-48e9-9718-a71057b3b615"
Oct 31 22:49:24.211339 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:24.211314    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ddxkp"]
Oct 31 22:49:24.211488 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:24.211427    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddxkp"
Oct 31 22:49:24.211535 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:24.211499    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddxkp" podUID="efec9075-8e74-475d-8160-48e356140743"
Oct 31 22:49:24.212169 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:24.212144    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5qkg4"]
Oct 31 22:49:24.212262 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:24.212249    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:24.212436 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:24.212408    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qkg4" podUID="81cc6a3e-e646-4287-84a7-d6830c40784b"
Oct 31 22:49:25.737799 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:25.737765    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddxkp"
Oct 31 22:49:25.738315 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:25.737780    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:25.738315 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:25.737879    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddxkp" podUID="efec9075-8e74-475d-8160-48e356140743"
Oct 31 22:49:25.738315 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:25.737780    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:25.738315 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:25.737948    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qkg4" podUID="81cc6a3e-e646-4287-84a7-d6830c40784b"
Oct 31 22:49:25.738315 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:25.738011    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv88p" podUID="620515c1-daf9-48e9-9718-a71057b3b615"
Oct 31 22:49:27.737881 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:27.737792    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:27.738234 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:27.737818    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:27.738234 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:27.737910    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv88p" podUID="620515c1-daf9-48e9-9718-a71057b3b615"
Oct 31 22:49:27.738234 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:27.737952    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddxkp"
Oct 31 22:49:27.738234 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:27.738033    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qkg4" podUID="81cc6a3e-e646-4287-84a7-d6830c40784b"
Oct 31 22:49:27.738234 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:27.738098    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddxkp" podUID="efec9075-8e74-475d-8160-48e356140743"
Oct 31 22:49:29.737591 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:29.737564    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddxkp"
Oct 31 22:49:29.737591 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:29.737582    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:29.738031 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:29.737671    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddxkp" podUID="efec9075-8e74-475d-8160-48e356140743"
Oct 31 22:49:29.738031 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:29.737740    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qkg4" podUID="81cc6a3e-e646-4287-84a7-d6830c40784b"
Oct 31 22:49:29.738031 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:29.737751    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:29.738031 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:29.737841    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv88p" podUID="620515c1-daf9-48e9-9718-a71057b3b615"
Oct 31 22:49:30.029199 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:30.029120    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-163.us-east-2.compute.internal" event="NodeReady"
Oct 31 22:49:30.029345 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:30.029239    2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 22:49:31.739080 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:31.738899    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddxkp"
Oct 31 22:49:31.739565 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:31.738977    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:31.739565 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:31.739002    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:31.743810 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:31.743786    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 22:49:31.746334 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:31.746314    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zbrj5\""
Oct 31 22:49:31.746465 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:31.746312    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-ssqvr\""
Oct 31 22:49:31.746465 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:31.746443    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 22:49:31.746579 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:31.746465    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 22:49:31.746579 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:31.746494    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-m4zzn\""
Oct 31 22:49:31.746579 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:31.746540    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 22:49:33.699479 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:33.699453    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cn8jx_e46ea70b-37a4-4934-920c-ecdab212ecc6/dns-node-resolver/0.log"
Oct 31 22:49:33.965791 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:33.965725    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vc2j7_5f102a31-27f2-410b-9e36-2cdb8d9e852c/konnectivity-agent/0.log" limit=1024
Oct 31 22:49:34.265204 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:34.265127    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vc2j7_5f102a31-27f2-410b-9e36-2cdb8d9e852c/konnectivity-agent/0.log" limit=1024
Oct 31 22:49:34.449261 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:34.449234    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vc2j7_5f102a31-27f2-410b-9e36-2cdb8d9e852c/konnectivity-agent/0.log" limit=1024
Oct 31 22:49:34.666443 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:34.666412    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vc2j7_5f102a31-27f2-410b-9e36-2cdb8d9e852c/konnectivity-agent/0.log" limit=1024
Oct 31 22:49:34.858538 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:34.858505    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vc2j7_5f102a31-27f2-410b-9e36-2cdb8d9e852c/konnectivity-agent/0.log" limit=1024
Oct 31 22:49:35.069351 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:35.069286    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xw8kl_8b1db31b-aec0-495b-8b0c-4abbde3acb28/node-ca/0.log"
Oct 31 22:49:35.524892 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:35.524855    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81cc6a3e-e646-4287-84a7-d6830c40784b-metrics-certs\") pod \"network-metrics-daemon-5qkg4\" (UID: \"81cc6a3e-e646-4287-84a7-d6830c40784b\") " pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:35.525091 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:35.525002    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 22:49:35.525091 ip-10-0-135-163 kubenswrapper[2575]: E1031 22:49:35.525074    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81cc6a3e-e646-4287-84a7-d6830c40784b-metrics-certs podName:81cc6a3e-e646-4287-84a7-d6830c40784b nodeName:}" failed. No retries permitted until 2026-10-31 22:50:07.525060622 +0000 UTC m=+66.283507061 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81cc6a3e-e646-4287-84a7-d6830c40784b-metrics-certs") pod "network-metrics-daemon-5qkg4" (UID: "81cc6a3e-e646-4287-84a7-d6830c40784b") : secret "metrics-daemon-secret" not found
Oct 31 22:49:35.625844 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:35.625807    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/620515c1-daf9-48e9-9718-a71057b3b615-original-pull-secret\") pod \"global-pull-secret-syncer-qv88p\" (UID: \"620515c1-daf9-48e9-9718-a71057b3b615\") " pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:35.625999 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:35.625852    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hqz8\" (UniqueName: \"kubernetes.io/projected/efec9075-8e74-475d-8160-48e356140743-kube-api-access-9hqz8\") pod \"network-check-target-ddxkp\" (UID: \"efec9075-8e74-475d-8160-48e356140743\") " pod="openshift-network-diagnostics/network-check-target-ddxkp"
Oct 31 22:49:35.629463 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:35.629433    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/620515c1-daf9-48e9-9718-a71057b3b615-original-pull-secret\") pod \"global-pull-secret-syncer-qv88p\" (UID: \"620515c1-daf9-48e9-9718-a71057b3b615\") " pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:35.629613 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:35.629596    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hqz8\" (UniqueName: \"kubernetes.io/projected/efec9075-8e74-475d-8160-48e356140743-kube-api-access-9hqz8\") pod \"network-check-target-ddxkp\" (UID: \"efec9075-8e74-475d-8160-48e356140743\") " pod="openshift-network-diagnostics/network-check-target-ddxkp"
Oct 31 22:49:35.695176 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:35.695148    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddxkp"
Oct 31 22:49:35.741256 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:35.741229    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:36.114598 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:36.114531    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ddxkp"]
Oct 31 22:49:36.117736 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:36.117672    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qv88p"]
Oct 31 22:49:36.118034 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:36.118006    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefec9075_8e74_475d_8160_48e356140743.slice/crio-55ff80fb1f1ede089ba94e10bdda200ea041a773ca473499ea65aa9eb5a606d3 WatchSource:0}: Error finding container 55ff80fb1f1ede089ba94e10bdda200ea041a773ca473499ea65aa9eb5a606d3: Status 404 returned error can't find the container with id 55ff80fb1f1ede089ba94e10bdda200ea041a773ca473499ea65aa9eb5a606d3
Oct 31 22:49:36.121552 ip-10-0-135-163 kubenswrapper[2575]: W1031 22:49:36.121483    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod620515c1_daf9_48e9_9718_a71057b3b615.slice/crio-ae9de0cb7f6fcc6e803269ca80675b8b420540a634f833c1b4d0c4d16f8d8d13 WatchSource:0}: Error finding container ae9de0cb7f6fcc6e803269ca80675b8b420540a634f833c1b4d0c4d16f8d8d13: Status 404 returned error can't find the container with id ae9de0cb7f6fcc6e803269ca80675b8b420540a634f833c1b4d0c4d16f8d8d13
Oct 31 22:49:37.009111 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:37.009062    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qv88p" event={"ID":"620515c1-daf9-48e9-9718-a71057b3b615","Type":"ContainerStarted","Data":"ae9de0cb7f6fcc6e803269ca80675b8b420540a634f833c1b4d0c4d16f8d8d13"}
Oct 31 22:49:37.010501 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:37.010448    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ddxkp" event={"ID":"efec9075-8e74-475d-8160-48e356140743","Type":"ContainerStarted","Data":"55ff80fb1f1ede089ba94e10bdda200ea041a773ca473499ea65aa9eb5a606d3"}
Oct 31 22:49:40.021950 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:40.021867    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ddxkp" event={"ID":"efec9075-8e74-475d-8160-48e356140743","Type":"ContainerStarted","Data":"70d75911a63b203618e74eb58753c13d1a881a4f7fb23a3a9726c6f93921c1d3"}
Oct 31 22:49:40.022417 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:40.022012    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ddxkp"
Oct 31 22:49:40.041766 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:40.041711    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ddxkp" podStartSLOduration=35.514306822 podStartE2EDuration="39.041697828s" totalImagesPullingTime="3.527391006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:49:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:40.041096866 +0000 UTC m=+38.799543309" watchObservedRunningTime="2026-10-31 22:49:40.041697828 +0000 UTC m=+38.800144267"
Oct 31 22:49:41.026192 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:41.026158    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qv88p" event={"ID":"620515c1-daf9-48e9-9718-a71057b3b615","Type":"ContainerStarted","Data":"9e1a892c97dc960219cf290f341fbc7a30a126810c56eb24e91145ed1d46284d"}
Oct 31 22:49:41.044699 ip-10-0-135-163 kubenswrapper[2575]: I1031 22:49:41.044597    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qv88p" podStartSLOduration=35.273724882 podStartE2EDuration="40.044578981s" totalImagesPullingTime="4.770854099s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:49:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:41.044393468 +0000 UTC m=+39.802839910" watchObservedRunningTime="2026-10-31 22:49:41.044578981 +0000 UTC m=+39.803025426"
Oct 31 22:49:41.197918 ip-10-0-135-163 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 22:49:41.209770 ip-10-0-135-163 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 22:49:41.209932 ip-10-0-135-163 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 22:49:41.209980 ip-10-0-135-163 systemd[1]: kubelet.service: Consumed 2.109s CPU time.
Oct 31 22:49:41.223950 ip-10-0-135-163 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 22:49:41.319376 ip-10-0-135-163 kubenswrapper[4301]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 22:49:41.319376 ip-10-0-135-163 kubenswrapper[4301]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 22:49:41.319376 ip-10-0-135-163 kubenswrapper[4301]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 22:49:41.319376 ip-10-0-135-163 kubenswrapper[4301]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 22:49:41.321875 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.321781    4301 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 22:49:41.321875 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.321870    4301 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 22:49:41.321875 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.321877    4301 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 22:49:41.321875 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.321881    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 22:49:41.321875 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.321886    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 22:49:41.322199 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.321890    4301 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 22:49:41.322199 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.321895    4301 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 22:49:41.322199 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.321903    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 22:49:41.322199 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.321907    4301 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 22:49:41.322199 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.321911    4301 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 22:49:41.322199 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.321917    4301 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 22:49:41.322199 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.321923    4301 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 22:49:41.322199 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.321928    4301 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 22:49:41.322199 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.321933    4301 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 22:49:41.322199 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.321937    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 22:49:41.322199 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.321941    4301 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 22:49:41.322199 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.321945    4301 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 22:49:41.322199 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.321949    4301 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 22:49:41.322199 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.321952    4301 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 22:49:41.322199 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.321956    4301 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 22:49:41.322199 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.321960    4301 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 22:49:41.322199 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.321963    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 22:49:41.322199 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.321968    4301 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 22:49:41.322199 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.321972    4301 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 22:49:41.323018 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.321975    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 22:49:41.323018 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.321979    4301 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 22:49:41.323018 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.321983    4301 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 22:49:41.323018 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.321988    4301 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 22:49:41.323018 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.321992    4301 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 22:49:41.323018 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.321996    4301 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 22:49:41.323018 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322001    4301 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 22:49:41.323018 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322006    4301 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 22:49:41.323018 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322011    4301 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 22:49:41.323018 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322015    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 22:49:41.323018 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322019    4301 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 22:49:41.323018 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322023    4301 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 22:49:41.323018 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322028    4301 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 22:49:41.323018 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322034    4301 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 22:49:41.323018 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322041    4301 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 22:49:41.323018 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322047    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 22:49:41.323018 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322052    4301 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 22:49:41.323018 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322056    4301 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 22:49:41.323018 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322063    4301 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 22:49:41.323858 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322067    4301 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 22:49:41.323858 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322071    4301 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 22:49:41.323858 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322075    4301 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 22:49:41.323858 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322079    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 22:49:41.323858 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322084    4301 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 22:49:41.323858 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322088    4301 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 22:49:41.323858 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322093    4301 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 22:49:41.323858 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322097    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 22:49:41.323858 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322104    4301 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 22:49:41.323858 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322108    4301 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 22:49:41.323858 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322112    4301 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 22:49:41.323858 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322116    4301 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 22:49:41.323858 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322121    4301 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 22:49:41.323858 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322125    4301 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 22:49:41.323858 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322129    4301 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 22:49:41.323858 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322134    4301 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 22:49:41.323858 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322138    4301 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 22:49:41.323858 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322142    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 22:49:41.323858 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322146    4301 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 22:49:41.323858 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322150    4301 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 22:49:41.324758 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322154    4301 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 22:49:41.324758 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322158    4301 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 22:49:41.324758 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322163    4301 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 22:49:41.324758 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322168    4301 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 22:49:41.324758 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322172    4301 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 22:49:41.324758 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322176    4301 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 22:49:41.324758 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322180    4301 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 22:49:41.324758 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322185    4301 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 22:49:41.324758 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322190    4301 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 22:49:41.324758 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322194    4301 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 22:49:41.324758 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322198    4301 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 22:49:41.324758 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322202    4301 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 22:49:41.324758 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322206    4301 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 22:49:41.324758 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322210    4301 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 22:49:41.324758 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322214    4301 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 22:49:41.324758 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322218    4301 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 22:49:41.324758 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322222    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 22:49:41.324758 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322226    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 22:49:41.324758 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322230    4301 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 22:49:41.324758 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322235    4301 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 22:49:41.325701 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322241    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 22:49:41.325701 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322245    4301 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 22:49:41.325701 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322249    4301 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 22:49:41.325701 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322254    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 22:49:41.325701 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322258    4301 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 22:49:41.325701 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322262    4301 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 22:49:41.325701 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322266    4301 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 22:49:41.325701 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322270    4301 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 22:49:41.325701 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322275    4301 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 22:49:41.325701 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322279    4301 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 22:49:41.325701 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322283    4301 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 22:49:41.325701 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322287    4301 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 22:49:41.325701 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322291    4301 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 22:49:41.325701 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322295    4301 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 22:49:41.325701 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322299    4301 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 22:49:41.325701 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322303    4301 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 22:49:41.325701 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322308    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 22:49:41.325701 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322312    4301 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 22:49:41.325701 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322316    4301 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 22:49:41.325701 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322321    4301 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 22:49:41.326592 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322325    4301 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 22:49:41.326592 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322330    4301 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 22:49:41.326592 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.322336    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 22:49:41.326592 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323632    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 22:49:41.326592 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323644    4301 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 22:49:41.326592 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323648    4301 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 22:49:41.326592 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323653    4301 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 22:49:41.326592 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323658    4301 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 22:49:41.326592 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323663    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 22:49:41.326592 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323667    4301 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 22:49:41.326592 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323671    4301 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 22:49:41.326592 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323678    4301 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 22:49:41.326592 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323684    4301 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 22:49:41.326592 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323690    4301 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 22:49:41.326592 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323696    4301 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 22:49:41.326592 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323702    4301 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 22:49:41.326592 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323706    4301 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 22:49:41.326592 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323711    4301 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 22:49:41.326592 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323715    4301 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 22:49:41.327426 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323719    4301 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 22:49:41.327426 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323723    4301 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 22:49:41.327426 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323727    4301 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 22:49:41.327426 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323731    4301 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 22:49:41.327426 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323735    4301 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 22:49:41.327426 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323742    4301 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 22:49:41.327426 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323748    4301 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 22:49:41.327426 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323753    4301 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 22:49:41.327426 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323757    4301 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 22:49:41.327426 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323762    4301 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 22:49:41.327426 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323766    4301 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 22:49:41.327426 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323770    4301 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 22:49:41.327426 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323775    4301 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 22:49:41.327426 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323779    4301 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 22:49:41.327426 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323784    4301 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 22:49:41.327426 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323789    4301 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 22:49:41.327426 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323793    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 22:49:41.327426 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323797    4301 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 22:49:41.327426 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323802    4301 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 22:49:41.328215 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323806    4301 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 22:49:41.328215 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323810    4301 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 22:49:41.328215 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323814    4301 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 22:49:41.328215 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323818    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 22:49:41.328215 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323823    4301 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 22:49:41.328215 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323827    4301 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 22:49:41.328215 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323831    4301 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 22:49:41.328215 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323836    4301 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 22:49:41.328215 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323840    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 22:49:41.328215 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323844    4301 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 22:49:41.328215 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323849    4301 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 22:49:41.328215 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323853    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 22:49:41.328215 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323857    4301 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 22:49:41.328215 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323862    4301 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 22:49:41.328215 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323865    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 22:49:41.328215 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323869    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 22:49:41.328215 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323873    4301 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 22:49:41.328215 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323877    4301 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 22:49:41.328215 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323881    4301 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 22:49:41.328215 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323885    4301 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 22:49:41.328793 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323889    4301 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 22:49:41.328793 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323894    4301 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 22:49:41.328793 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323898    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 22:49:41.328793 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323903    4301 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 22:49:41.328793 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323907    4301 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 22:49:41.328793 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323912    4301 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 22:49:41.328793 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323917    4301 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 22:49:41.328793 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323921    4301 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 22:49:41.328793 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323925    4301 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 22:49:41.328793 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323930    4301 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 22:49:41.328793 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323934    4301 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 22:49:41.328793 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323938    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 22:49:41.328793 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323943    4301 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 22:49:41.328793 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323947    4301 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 22:49:41.328793 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323952    4301 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 22:49:41.328793 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323956    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 22:49:41.328793 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323960    4301 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 22:49:41.328793 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323965    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 22:49:41.328793 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323969    4301 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 22:49:41.329262 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323973    4301 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 22:49:41.329262 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323977    4301 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 22:49:41.329262 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323981    4301 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 22:49:41.329262 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323985    4301 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 22:49:41.329262 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323990    4301 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 22:49:41.329262 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323994    4301 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 22:49:41.329262 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.323998    4301 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 22:49:41.329262 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.324002    4301 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 22:49:41.329262 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.324006    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 22:49:41.329262 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.324011    4301 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 22:49:41.329262 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.324015    4301 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 22:49:41.329262 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.324019    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 22:49:41.329262 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.324023    4301 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 22:49:41.329262 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.324027    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 22:49:41.329262 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.324032    4301 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 22:49:41.329262 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.324036    4301 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 22:49:41.329262 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.324040    4301 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 22:49:41.329262 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.324044    4301 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 22:49:41.329262 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.324048    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 22:49:41.329755 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.324053    4301 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 22:49:41.329755 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.324058    4301 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 22:49:41.329755 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.324062    4301 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 22:49:41.329755 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.324066    4301 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 22:49:41.329755 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.324071    4301 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 22:49:41.329755 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.324075    4301 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 22:49:41.329755 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.324080    4301 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 22:49:41.329755 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.324084    4301 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 22:49:41.329755 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.324088    4301 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 22:49:41.329755 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.324094    4301 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 22:49:41.329755 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.324098    4301 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 22:49:41.329755 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.324102    4301 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 22:49:41.329755 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.324106    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 22:49:41.329755 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.324972    4301 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 22:49:41.329755 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.324982    4301 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 22:49:41.329755 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.324989    4301 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 22:49:41.329755 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.324994    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 22:49:41.329755 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.324998    4301 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 22:49:41.329755 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325005    4301 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 22:49:41.329755 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325010    4301 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 22:49:41.330246 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325014    4301 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 22:49:41.330246 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325019    4301 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 22:49:41.330246 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325023    4301 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 22:49:41.330246 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325028    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 22:49:41.330246 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325032    4301 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 22:49:41.330246 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325036    4301 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 22:49:41.330246 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325040    4301 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 22:49:41.330246 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325045    4301 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 22:49:41.330246 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325049    4301 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 22:49:41.330246 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325054    4301 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 22:49:41.330246 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325058    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 22:49:41.330246 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325063    4301 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 22:49:41.330246 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325069    4301 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 22:49:41.330246 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325076    4301 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 22:49:41.330246 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325081    4301 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 22:49:41.330246 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325087    4301 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 22:49:41.330246 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325092    4301 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 22:49:41.330246 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325097    4301 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 22:49:41.330246 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325102    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 22:49:41.330733 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325106    4301 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 22:49:41.330733 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325110    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 22:49:41.330733 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325115    4301 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 22:49:41.330733 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325119    4301 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 22:49:41.330733 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325123    4301 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 22:49:41.330733 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325127    4301 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 22:49:41.330733 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325132    4301 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 22:49:41.330733 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325136    4301 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 22:49:41.330733 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325141    4301 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 22:49:41.330733 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325145    4301 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 22:49:41.330733 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325150    4301 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 22:49:41.330733 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325156    4301 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 22:49:41.330733 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325162    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 22:49:41.330733 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325166    4301 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 22:49:41.330733 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325170    4301 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 22:49:41.330733 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325175    4301 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 22:49:41.330733 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325179    4301 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 22:49:41.330733 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325183    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 22:49:41.330733 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325187    4301 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 22:49:41.330733 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325192    4301 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 22:49:41.331203 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325196    4301 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 22:49:41.331203 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325200    4301 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 22:49:41.331203 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325205    4301 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 22:49:41.331203 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325209    4301 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 22:49:41.331203 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325214    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 22:49:41.331203 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325218    4301 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 22:49:41.331203 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325222    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 22:49:41.331203 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325226    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 22:49:41.331203 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325231    4301 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 22:49:41.331203 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325235    4301 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 22:49:41.331203 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325240    4301 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 22:49:41.331203 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325245    4301 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 22:49:41.331203 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325249    4301 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 22:49:41.331203 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325254    4301 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 22:49:41.331203 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325258    4301 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 22:49:41.331203 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325262    4301 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 22:49:41.331203 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325266    4301 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 22:49:41.331203 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325270    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 22:49:41.331203 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325275    4301 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 22:49:41.331203 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325279    4301 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 22:49:41.331688 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325283    4301 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 22:49:41.331688 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325287    4301 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 22:49:41.331688 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325291    4301 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 22:49:41.331688 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325298    4301 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 22:49:41.331688 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325302    4301 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 22:49:41.331688 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325306    4301 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 22:49:41.331688 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325310    4301 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 22:49:41.331688 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325315    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 22:49:41.331688 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325319    4301 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 22:49:41.331688 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325323    4301 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 22:49:41.331688 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325327    4301 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 22:49:41.331688 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325332    4301 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 22:49:41.331688 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325336    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 22:49:41.331688 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325340    4301 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 22:49:41.331688 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325344    4301 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 22:49:41.331688 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325349    4301 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 22:49:41.331688 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325368    4301 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 22:49:41.331688 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325373    4301 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 22:49:41.331688 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325378    4301 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 22:49:41.332139 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325382    4301 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 22:49:41.332139 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325386    4301 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 22:49:41.332139 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325390    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 22:49:41.332139 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325395    4301 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 22:49:41.332139 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325399    4301 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 22:49:41.332139 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325404    4301 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 22:49:41.332139 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325408    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 22:49:41.332139 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325413    4301 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 22:49:41.332139 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325418    4301 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 22:49:41.332139 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325422    4301 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 22:49:41.332139 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325430    4301 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 22:49:41.332139 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325436    4301 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 22:49:41.332139 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325441    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 22:49:41.332139 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325445    4301 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 22:49:41.332139 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325449    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 22:49:41.332139 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325454    4301 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 22:49:41.332139 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325461    4301 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 22:49:41.332139 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325465    4301 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 22:49:41.332139 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325469    4301 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 22:49:41.332653 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325474    4301 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 22:49:41.332653 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.325478    4301 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 22:49:41.332653 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325634    4301 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 22:49:41.332653 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325645    4301 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 22:49:41.332653 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325656    4301 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 22:49:41.332653 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325663    4301 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 22:49:41.332653 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325670    4301 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 22:49:41.332653 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325675    4301 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 22:49:41.332653 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325682    4301 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 22:49:41.332653 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325689    4301 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 22:49:41.332653 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325694    4301 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 22:49:41.332653 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325698    4301 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 22:49:41.332653 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325704    4301 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 22:49:41.332653 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325709    4301 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 22:49:41.332653 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325714    4301 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 22:49:41.332653 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325718    4301 flags.go:64] FLAG: --cgroup-root=""
Oct 31 22:49:41.332653 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325723    4301 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 22:49:41.332653 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325728    4301 flags.go:64] FLAG: --client-ca-file=""
Oct 31 22:49:41.332653 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325733    4301 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 22:49:41.332653 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325739    4301 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 22:49:41.332653 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325745    4301 flags.go:64] FLAG: --cluster-domain=""
Oct 31 22:49:41.332653 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325750    4301 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 22:49:41.332653 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325755    4301 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 22:49:41.332653 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325760    4301 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 22:49:41.333225 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325765    4301 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 22:49:41.333225 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325772    4301 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 22:49:41.333225 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325777    4301 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 22:49:41.333225 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325782    4301 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 22:49:41.333225 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325787    4301 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 22:49:41.333225 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325794    4301 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 22:49:41.333225 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325799    4301 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 22:49:41.333225 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325804    4301 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 22:49:41.333225 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325808    4301 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 22:49:41.333225 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325813    4301 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 22:49:41.333225 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325820    4301 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 22:49:41.333225 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325825    4301 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 22:49:41.333225 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325830    4301 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 22:49:41.333225 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325835    4301 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 22:49:41.333225 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325840    4301 flags.go:64] FLAG: --enable-server="true"
Oct 31 22:49:41.333225 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325845    4301 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 22:49:41.333225 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325852    4301 flags.go:64] FLAG: --event-burst="100"
Oct 31 22:49:41.333225 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325857    4301 flags.go:64] FLAG: --event-qps="50"
Oct 31 22:49:41.333225 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325862    4301 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 22:49:41.333225 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325867    4301 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 22:49:41.333225 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325872    4301 flags.go:64] FLAG: --eviction-hard=""
Oct 31 22:49:41.333225 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325880    4301 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 22:49:41.333225 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325884    4301 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 22:49:41.333225 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325889    4301 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 22:49:41.333225 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325894    4301 flags.go:64] FLAG: --eviction-soft=""
Oct 31 22:49:41.333907 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325899    4301 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 22:49:41.333907 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325904    4301 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 22:49:41.333907 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325910    4301 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 22:49:41.333907 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325915    4301 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 22:49:41.333907 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325920    4301 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 22:49:41.333907 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325925    4301 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 22:49:41.333907 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325929    4301 flags.go:64] FLAG: --feature-gates=""
Oct 31 22:49:41.333907 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325936    4301 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 22:49:41.333907 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325942    4301 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 22:49:41.333907 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325946    4301 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 22:49:41.333907 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325952    4301 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 22:49:41.333907 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325957    4301 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 22:49:41.333907 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325963    4301 flags.go:64] FLAG: --help="false"
Oct 31 22:49:41.333907 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325969    4301 flags.go:64] FLAG: --hostname-override="ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:41.333907 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325974    4301 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 22:49:41.333907 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325979    4301 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 22:49:41.333907 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325983    4301 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 22:49:41.333907 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325989    4301 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 22:49:41.333907 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.325995    4301 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 22:49:41.333907 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326000    4301 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 22:49:41.333907 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326004    4301 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 22:49:41.333907 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326009    4301 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 22:49:41.333907 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326016    4301 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 22:49:41.334475 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326021    4301 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 22:49:41.334475 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326026    4301 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 22:49:41.334475 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326031    4301 flags.go:64] FLAG: --kube-reserved=""
Oct 31 22:49:41.334475 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326036    4301 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 22:49:41.334475 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326041    4301 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 22:49:41.334475 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326046    4301 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 22:49:41.334475 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326051    4301 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 22:49:41.334475 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326056    4301 flags.go:64] FLAG: --lock-file=""
Oct 31 22:49:41.334475 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326060    4301 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 22:49:41.334475 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326065    4301 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 22:49:41.334475 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326070    4301 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 22:49:41.334475 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326079    4301 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 22:49:41.334475 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326084    4301 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 22:49:41.334475 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326089    4301 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 22:49:41.334475 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326094    4301 flags.go:64] FLAG: --logging-format="text"
Oct 31 22:49:41.334475 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326099    4301 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 22:49:41.334475 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326104    4301 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 22:49:41.334475 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326109    4301 flags.go:64] FLAG: --manifest-url=""
Oct 31 22:49:41.334475 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326113    4301 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 22:49:41.334475 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326120    4301 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 22:49:41.334475 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326126    4301 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 22:49:41.334475 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326133    4301 flags.go:64] FLAG: --max-pods="110"
Oct 31 22:49:41.334475 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326138    4301 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 22:49:41.334475 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326143    4301 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 22:49:41.334475 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326148    4301 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 22:49:41.335075 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326152    4301 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 22:49:41.335075 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326157    4301 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 22:49:41.335075 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326162    4301 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 22:49:41.335075 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326168    4301 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 22:49:41.335075 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326179    4301 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 22:49:41.335075 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326185    4301 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 22:49:41.335075 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326191    4301 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 22:49:41.335075 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326196    4301 flags.go:64] FLAG: --pod-cidr=""
Oct 31 22:49:41.335075 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326201    4301 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 22:49:41.335075 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326205    4301 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 22:49:41.335075 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326211    4301 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 22:49:41.335075 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326215    4301 flags.go:64] FLAG: --port="10250"
Oct 31 22:49:41.335075 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326220    4301 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 22:49:41.335075 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326225    4301 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0d2c56eb0f44aa2ca"
Oct 31 22:49:41.335075 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326230    4301 flags.go:64] FLAG: --qos-reserved=""
Oct 31 22:49:41.335075 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326235    4301 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 22:49:41.335075 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326240    4301 flags.go:64] FLAG: --register-node="true"
Oct 31 22:49:41.335075 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326244    4301 flags.go:64] FLAG: --register-with-taints=""
Oct 31 22:49:41.335075 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326250    4301 flags.go:64] FLAG: --registry-burst="10"
Oct 31 22:49:41.335075 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326255    4301 flags.go:64] FLAG: --registry-qps="5"
Oct 31 22:49:41.335075 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326261    4301 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 22:49:41.335075 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326266    4301 flags.go:64] FLAG: --reserved-memory=""
Oct 31 22:49:41.335075 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326272    4301 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 22:49:41.335075 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326276    4301 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 22:49:41.335075 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326281    4301 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 22:49:41.335075 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326286    4301 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 22:49:41.335696 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326291    4301 flags.go:64] FLAG: --runonce="false"
Oct 31 22:49:41.335696 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326295    4301 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 22:49:41.335696 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326302    4301 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 22:49:41.335696 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326307    4301 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 22:49:41.335696 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326312    4301 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 22:49:41.335696 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326317    4301 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 22:49:41.335696 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326322    4301 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 22:49:41.335696 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326326    4301 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 22:49:41.335696 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326331    4301 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 22:49:41.335696 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326336    4301 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 22:49:41.335696 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326341    4301 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 22:49:41.335696 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326345    4301 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 22:49:41.335696 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326352    4301 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 22:49:41.335696 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326373    4301 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 22:49:41.335696 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326378    4301 flags.go:64] FLAG: --system-cgroups=""
Oct 31 22:49:41.335696 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326383    4301 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 22:49:41.335696 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326392    4301 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 22:49:41.335696 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326397    4301 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 22:49:41.335696 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326401    4301 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 22:49:41.335696 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326407    4301 flags.go:64] FLAG: --tls-min-version=""
Oct 31 22:49:41.335696 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326412    4301 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 22:49:41.335696 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326417    4301 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 22:49:41.335696 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326421    4301 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 22:49:41.335696 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326426    4301 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 22:49:41.335696 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326431    4301 flags.go:64] FLAG: --v="2"
Oct 31 22:49:41.336280 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326438    4301 flags.go:64] FLAG: --version="false"
Oct 31 22:49:41.336280 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326445    4301 flags.go:64] FLAG: --vmodule=""
Oct 31 22:49:41.336280 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326452    4301 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 22:49:41.336280 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.326457    4301 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 22:49:41.336280 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.326833    4301 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 22:49:41.336280 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.326856    4301 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 22:49:41.336280 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.326864    4301 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 22:49:41.336280 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.326878    4301 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 22:49:41.336280 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.326884    4301 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 22:49:41.336280 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.326893    4301 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 22:49:41.336280 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.326899    4301 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 22:49:41.336280 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.326903    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 22:49:41.336280 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.326908    4301 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 22:49:41.336280 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.326912    4301 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 22:49:41.336280 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.326917    4301 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 22:49:41.336280 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.326921    4301 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 22:49:41.336280 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.326925    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 22:49:41.336280 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.326930    4301 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 22:49:41.336280 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.326935    4301 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 22:49:41.336280 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.326941    4301 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 22:49:41.336923 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.326951    4301 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 22:49:41.336923 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.326955    4301 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 22:49:41.336923 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.326960    4301 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 22:49:41.336923 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.326964    4301 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 22:49:41.336923 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.326968    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 22:49:41.336923 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.326973    4301 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 22:49:41.336923 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.326978    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 22:49:41.336923 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.326982    4301 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 22:49:41.336923 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.326987    4301 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 22:49:41.336923 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.326991    4301 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 22:49:41.336923 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.326996    4301 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 22:49:41.336923 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327000    4301 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 22:49:41.336923 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327005    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 22:49:41.336923 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327015    4301 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 22:49:41.336923 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327019    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 22:49:41.336923 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327025    4301 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 22:49:41.336923 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327029    4301 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 22:49:41.336923 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327033    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 22:49:41.336923 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327038    4301 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 22:49:41.337681 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327043    4301 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 22:49:41.337681 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327047    4301 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 22:49:41.337681 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327053    4301 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 22:49:41.337681 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327058    4301 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 22:49:41.337681 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327063    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 22:49:41.337681 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327067    4301 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 22:49:41.337681 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327076    4301 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 22:49:41.337681 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327080    4301 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 22:49:41.337681 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327085    4301 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 22:49:41.337681 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327089    4301 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 22:49:41.337681 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327093    4301 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 22:49:41.337681 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327098    4301 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 22:49:41.337681 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327104    4301 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 22:49:41.337681 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327108    4301 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 22:49:41.337681 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327113    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 22:49:41.337681 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327118    4301 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 22:49:41.337681 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327122    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 22:49:41.337681 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327127    4301 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 22:49:41.337681 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327132    4301 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 22:49:41.337681 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327140    4301 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 22:49:41.338393 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327145    4301 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 22:49:41.338393 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327149    4301 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 22:49:41.338393 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327153    4301 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 22:49:41.338393 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327158    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 22:49:41.338393 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327163    4301 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 22:49:41.338393 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327167    4301 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 22:49:41.338393 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327172    4301 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 22:49:41.338393 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327176    4301 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 22:49:41.338393 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327180    4301 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 22:49:41.338393 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327186    4301 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 22:49:41.338393 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327190    4301 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 22:49:41.338393 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327195    4301 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 22:49:41.338393 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327204    4301 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 22:49:41.338393 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327209    4301 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 22:49:41.338393 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327215    4301 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 22:49:41.338393 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327219    4301 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 22:49:41.338393 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327223    4301 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 22:49:41.338393 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327228    4301 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 22:49:41.338393 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327232    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 22:49:41.338393 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327236    4301 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 22:49:41.339078 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327241    4301 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 22:49:41.339078 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327245    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 22:49:41.339078 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327250    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 22:49:41.339078 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327254    4301 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 22:49:41.339078 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327265    4301 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 22:49:41.339078 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327269    4301 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 22:49:41.339078 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327274    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 22:49:41.339078 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327278    4301 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 22:49:41.339078 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327283    4301 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 22:49:41.339078 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327288    4301 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 22:49:41.339078 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327294    4301 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 22:49:41.339078 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327302    4301 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 22:49:41.339078 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327307    4301 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 22:49:41.339078 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327311    4301 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 22:49:41.339078 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327316    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 22:49:41.339078 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327320    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 22:49:41.339078 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327329    4301 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 22:49:41.339078 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327334    4301 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 22:49:41.339078 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327339    4301 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 22:49:41.340060 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327343    4301 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 22:49:41.340060 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327348    4301 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 22:49:41.340060 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327352    4301 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 22:49:41.340060 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327373    4301 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 22:49:41.340060 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327378    4301 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 22:49:41.340060 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327382    4301 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 22:49:41.340060 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327387    4301 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 22:49:41.340060 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327395    4301 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 22:49:41.340060 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327400    4301 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 22:49:41.340060 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327405    4301 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 22:49:41.340060 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327414    4301 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 22:49:41.340060 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327418    4301 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 22:49:41.340060 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.327476    4301 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 22:49:41.340060 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327608    4301 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 22:49:41.340060 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327619    4301 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 22:49:41.340060 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327625    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 22:49:41.340060 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327631    4301 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 22:49:41.341009 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327639    4301 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 22:49:41.341009 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327644    4301 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 22:49:41.341009 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327648    4301 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 22:49:41.341009 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327653    4301 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 22:49:41.341009 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327658    4301 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 22:49:41.341009 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327662    4301 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 22:49:41.341009 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327672    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 22:49:41.341009 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327678    4301 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 22:49:41.341009 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327682    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 22:49:41.341009 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327687    4301 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 22:49:41.341009 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327692    4301 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 22:49:41.341009 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327696    4301 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 22:49:41.341009 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327701    4301 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 22:49:41.341009 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327705    4301 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 22:49:41.341009 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327710    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 22:49:41.341009 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327714    4301 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 22:49:41.341009 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327719    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 22:49:41.341009 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327723    4301 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 22:49:41.341009 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327728    4301 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 22:49:41.341667 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327740    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 22:49:41.341667 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327744    4301 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 22:49:41.341667 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327749    4301 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 22:49:41.341667 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327754    4301 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 22:49:41.341667 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327759    4301 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 22:49:41.341667 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327763    4301 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 22:49:41.341667 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327768    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 22:49:41.341667 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327772    4301 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 22:49:41.341667 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327776    4301 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 22:49:41.341667 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327781    4301 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 22:49:41.341667 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327785    4301 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 22:49:41.341667 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327790    4301 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 22:49:41.341667 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327799    4301 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 22:49:41.341667 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327806    4301 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 22:49:41.341667 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327810    4301 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 22:49:41.341667 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327814    4301 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 22:49:41.341667 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327819    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 22:49:41.341667 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327824    4301 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 22:49:41.341667 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327828    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 22:49:41.341667 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327832    4301 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 22:49:41.342184 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327837    4301 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 22:49:41.342184 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327841    4301 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 22:49:41.342184 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327845    4301 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 22:49:41.342184 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327850    4301 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 22:49:41.342184 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327854    4301 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 22:49:41.342184 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327864    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 22:49:41.342184 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327869    4301 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 22:49:41.342184 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327873    4301 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 22:49:41.342184 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327878    4301 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 22:49:41.342184 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327882    4301 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 22:49:41.342184 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327886    4301 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 22:49:41.342184 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327890    4301 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 22:49:41.342184 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327895    4301 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 22:49:41.342184 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327900    4301 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 22:49:41.342184 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327905    4301 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 22:49:41.342184 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327909    4301 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 22:49:41.342184 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327913    4301 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 22:49:41.342184 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327918    4301 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 22:49:41.342184 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327927    4301 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 22:49:41.342184 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327931    4301 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 22:49:41.342703 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327935    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 22:49:41.342703 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327940    4301 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 22:49:41.342703 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327944    4301 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 22:49:41.342703 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327948    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 22:49:41.342703 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327953    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 22:49:41.342703 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327959    4301 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 22:49:41.342703 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327964    4301 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 22:49:41.342703 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327970    4301 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 22:49:41.342703 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327974    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 22:49:41.342703 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327979    4301 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 22:49:41.342703 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327988    4301 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 22:49:41.342703 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.327993    4301 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 22:49:41.342703 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.328000    4301 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 22:49:41.342703 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.328007    4301 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 22:49:41.342703 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.328018    4301 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 22:49:41.342703 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.328023    4301 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 22:49:41.342703 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.328150    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 22:49:41.342703 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.328158    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 22:49:41.342703 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.328162    4301 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 22:49:41.343149 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.328164    4301 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 22:49:41.343149 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.328167    4301 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 22:49:41.343149 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.328170    4301 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 22:49:41.343149 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.328173    4301 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 22:49:41.343149 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.328176    4301 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 22:49:41.343149 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.328179    4301 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 22:49:41.343149 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.328182    4301 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 22:49:41.343149 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.328186    4301 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 22:49:41.343149 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.328188    4301 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 22:49:41.343149 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.328191    4301 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 22:49:41.343149 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.328194    4301 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 22:49:41.343149 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.328197    4301 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 22:49:41.343149 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.328200    4301 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 22:49:41.343149 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.328202    4301 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 22:49:41.343149 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.328205    4301 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 22:49:41.343149 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.328208    4301 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 22:49:41.343149 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.328213    4301 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 22:49:41.343149 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.328217    4301 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 22:49:41.343149 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.328221    4301 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 22:49:41.343149 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.328224    4301 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 22:49:41.343745 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.328229    4301 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 22:49:41.343745 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.328231    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 22:49:41.343745 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.328237    4301 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 22:49:41.343745 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.328240    4301 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 22:49:41.343745 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.332134    4301 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 22:49:41.343745 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.332151    4301 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 22:49:41.343745 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.332169    4301 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 22:49:41.343745 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.332174    4301 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 22:49:41.343745 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332231    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 22:49:41.343745 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332237    4301 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 22:49:41.343745 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332241    4301 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 22:49:41.343745 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332245    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 22:49:41.343745 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332249    4301 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 22:49:41.343745 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332251    4301 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 22:49:41.343745 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332254    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 22:49:41.343745 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332257    4301 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 22:49:41.343745 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332260    4301 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 22:49:41.343745 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332263    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 22:49:41.343745 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332266    4301 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 22:49:41.344231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332268    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 22:49:41.344231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332271    4301 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 22:49:41.344231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332274    4301 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 22:49:41.344231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332277    4301 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 22:49:41.344231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332279    4301 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 22:49:41.344231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332282    4301 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 22:49:41.344231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332285    4301 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 22:49:41.344231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332287    4301 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 22:49:41.344231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332290    4301 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 22:49:41.344231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332293    4301 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 22:49:41.344231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332295    4301 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 22:49:41.344231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332298    4301 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 22:49:41.344231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332300    4301 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 22:49:41.344231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332303    4301 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 22:49:41.344231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332306    4301 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 22:49:41.344231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332309    4301 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 22:49:41.344231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332312    4301 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 22:49:41.344231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332316    4301 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 22:49:41.344231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332328    4301 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 22:49:41.344737 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332334    4301 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 22:49:41.344737 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332337    4301 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 22:49:41.344737 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332340    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 22:49:41.344737 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332343    4301 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 22:49:41.344737 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332346    4301 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 22:49:41.344737 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332348    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 22:49:41.344737 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332351    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 22:49:41.344737 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332390    4301 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 22:49:41.344737 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332395    4301 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 22:49:41.344737 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332397    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 22:49:41.344737 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332400    4301 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 22:49:41.344737 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332403    4301 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 22:49:41.344737 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332407    4301 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 22:49:41.344737 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332410    4301 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 22:49:41.344737 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332413    4301 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 22:49:41.344737 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332416    4301 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 22:49:41.344737 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332419    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 22:49:41.344737 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332422    4301 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 22:49:41.344737 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332425    4301 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 22:49:41.344737 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332427    4301 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 22:49:41.345335 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332430    4301 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 22:49:41.345335 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332433    4301 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 22:49:41.345335 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332435    4301 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 22:49:41.345335 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332438    4301 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 22:49:41.345335 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332441    4301 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 22:49:41.345335 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332443    4301 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 22:49:41.345335 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332446    4301 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 22:49:41.345335 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332448    4301 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 22:49:41.345335 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332451    4301 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 22:49:41.345335 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332453    4301 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 22:49:41.345335 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332456    4301 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 22:49:41.345335 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332459    4301 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 22:49:41.345335 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332462    4301 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 22:49:41.345335 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332465    4301 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 22:49:41.345335 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332468    4301 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 22:49:41.345335 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332471    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 22:49:41.345335 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332474    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 22:49:41.345335 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332476    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 22:49:41.345335 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332479    4301 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 22:49:41.345335 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332481    4301 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 22:49:41.346133 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332484    4301 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 22:49:41.346133 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332487    4301 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 22:49:41.346133 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332490    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 22:49:41.346133 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332492    4301 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 22:49:41.346133 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332495    4301 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 22:49:41.346133 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332498    4301 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 22:49:41.346133 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332500    4301 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 22:49:41.346133 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332503    4301 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 22:49:41.346133 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332505    4301 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 22:49:41.346133 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332508    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 22:49:41.346133 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332511    4301 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 22:49:41.346133 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332513    4301 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 22:49:41.346133 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332515    4301 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 22:49:41.346133 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332518    4301 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 22:49:41.346133 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332521    4301 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 22:49:41.346133 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332523    4301 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 22:49:41.346133 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332527    4301 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 22:49:41.346133 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332529    4301 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 22:49:41.346133 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332532    4301 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 22:49:41.346840 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332534    4301 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 22:49:41.346840 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332537    4301 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 22:49:41.346840 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332540    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 22:49:41.346840 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332543    4301 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 22:49:41.346840 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332545    4301 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 22:49:41.346840 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332548    4301 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 22:49:41.346840 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332550    4301 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 22:49:41.346840 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332553    4301 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 22:49:41.346840 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332556    4301 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 22:49:41.346840 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332559    4301 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 22:49:41.346840 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332562    4301 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 22:49:41.346840 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332564    4301 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 22:49:41.346840 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332567    4301 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 22:49:41.346840 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332570    4301 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 22:49:41.346840 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332572    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 22:49:41.346840 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332575    4301 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 22:49:41.346840 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332577    4301 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 22:49:41.347248 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.332606    4301 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 22:49:41.347248 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332770    4301 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 22:49:41.347248 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332776    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 22:49:41.347248 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332781    4301 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 22:49:41.347248 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332786    4301 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 22:49:41.347248 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332789    4301 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 22:49:41.347248 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332792    4301 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 22:49:41.347248 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332795    4301 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 22:49:41.347248 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332798    4301 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 22:49:41.347248 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332801    4301 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 22:49:41.347248 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332804    4301 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 22:49:41.347248 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332806    4301 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 22:49:41.347248 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332809    4301 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 22:49:41.347248 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332813    4301 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 22:49:41.347248 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332815    4301 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 22:49:41.347632 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332818    4301 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 22:49:41.347632 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332820    4301 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 22:49:41.347632 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332823    4301 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 22:49:41.347632 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332825    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 22:49:41.347632 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332828    4301 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 22:49:41.347632 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332831    4301 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 22:49:41.347632 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332834    4301 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 22:49:41.347632 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332836    4301 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 22:49:41.347632 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332840    4301 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 22:49:41.347632 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332843    4301 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 22:49:41.347632 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332845    4301 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 22:49:41.347632 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332848    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 22:49:41.347632 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332851    4301 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 22:49:41.347632 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332853    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 22:49:41.347632 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332856    4301 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 22:49:41.347632 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332859    4301 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 22:49:41.347632 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332862    4301 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 22:49:41.347632 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332864    4301 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 22:49:41.347632 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332867    4301 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 22:49:41.347632 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332869    4301 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 22:49:41.348104 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332872    4301 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 22:49:41.348104 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332875    4301 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 22:49:41.348104 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332879    4301 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 22:49:41.348104 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332882    4301 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 22:49:41.348104 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332885    4301 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 22:49:41.348104 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332888    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 22:49:41.348104 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332891    4301 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 22:49:41.348104 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332893    4301 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 22:49:41.348104 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332896    4301 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 22:49:41.348104 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332898    4301 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 22:49:41.348104 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332901    4301 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 22:49:41.348104 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332904    4301 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 22:49:41.348104 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332906    4301 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 22:49:41.348104 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332910    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 22:49:41.348104 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332912    4301 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 22:49:41.348104 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332915    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 22:49:41.348104 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332918    4301 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 22:49:41.348104 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332921    4301 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 22:49:41.348104 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332924    4301 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 22:49:41.348660 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332926    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 22:49:41.348660 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332929    4301 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 22:49:41.348660 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332932    4301 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 22:49:41.348660 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332935    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 22:49:41.348660 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332937    4301 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 22:49:41.348660 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332940    4301 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 22:49:41.348660 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332942    4301 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 22:49:41.348660 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332945    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 22:49:41.348660 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332947    4301 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 22:49:41.348660 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332950    4301 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 22:49:41.348660 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332952    4301 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 22:49:41.348660 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332955    4301 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 22:49:41.348660 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332957    4301 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 22:49:41.348660 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332960    4301 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 22:49:41.348660 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332963    4301 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 22:49:41.348660 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332965    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 22:49:41.348660 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332968    4301 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 22:49:41.348660 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332971    4301 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 22:49:41.348660 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332973    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 22:49:41.348660 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332976    4301 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 22:49:41.349166 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332978    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 22:49:41.349166 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332981    4301 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 22:49:41.349166 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332984    4301 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 22:49:41.349166 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332986    4301 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 22:49:41.349166 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332989    4301 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 22:49:41.349166 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332992    4301 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 22:49:41.349166 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332994    4301 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 22:49:41.349166 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332997    4301 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 22:49:41.349166 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.332999    4301 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 22:49:41.349166 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333002    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 22:49:41.349166 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333005    4301 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 22:49:41.349166 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333008    4301 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 22:49:41.349166 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333010    4301 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 22:49:41.349166 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333013    4301 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 22:49:41.349166 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333016    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 22:49:41.349166 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333019    4301 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 22:49:41.349166 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333021    4301 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 22:49:41.349166 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333024    4301 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 22:49:41.349166 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333027    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 22:49:41.349796 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333029    4301 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 22:49:41.349796 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333032    4301 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 22:49:41.349796 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333034    4301 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 22:49:41.349796 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333037    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 22:49:41.349796 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333039    4301 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 22:49:41.349796 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333041    4301 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 22:49:41.349796 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333044    4301 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 22:49:41.349796 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333047    4301 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 22:49:41.349796 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333052    4301 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 22:49:41.349796 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333055    4301 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 22:49:41.349796 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333057    4301 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 22:49:41.349796 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333060    4301 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 22:49:41.349796 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333063    4301 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 22:49:41.349796 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333065    4301 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 22:49:41.349796 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.333088    4301 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 22:49:41.349796 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333124    4301 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 22:49:41.350231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333128    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 22:49:41.350231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333131    4301 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 22:49:41.350231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333134    4301 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 22:49:41.350231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333137    4301 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 22:49:41.350231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333140    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 22:49:41.350231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333143    4301 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 22:49:41.350231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333145    4301 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 22:49:41.350231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333148    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 22:49:41.350231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333151    4301 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 22:49:41.350231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333153    4301 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 22:49:41.350231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333156    4301 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 22:49:41.350231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333158    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 22:49:41.350231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333161    4301 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 22:49:41.350231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333164    4301 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 22:49:41.350231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333166    4301 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 22:49:41.350231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333169    4301 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 22:49:41.350231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333172    4301 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 22:49:41.350231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333175    4301 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 22:49:41.350231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333178    4301 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 22:49:41.350231 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333180    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 22:49:41.350882 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333183    4301 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 22:49:41.350882 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333185    4301 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 22:49:41.350882 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333189    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 22:49:41.350882 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333191    4301 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 22:49:41.350882 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333194    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 22:49:41.350882 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333200    4301 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 22:49:41.350882 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333202    4301 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 22:49:41.350882 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333205    4301 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 22:49:41.350882 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333208    4301 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 22:49:41.350882 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333210    4301 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 22:49:41.350882 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333213    4301 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 22:49:41.350882 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333215    4301 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 22:49:41.350882 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333218    4301 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 22:49:41.350882 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333220    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 22:49:41.350882 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333223    4301 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 22:49:41.350882 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333225    4301 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 22:49:41.350882 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333228    4301 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 22:49:41.350882 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333231    4301 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 22:49:41.350882 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333233    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 22:49:41.351910 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333236    4301 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 22:49:41.351910 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333239    4301 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 22:49:41.351910 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333241    4301 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 22:49:41.351910 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333244    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 22:49:41.351910 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333247    4301 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 22:49:41.351910 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333249    4301 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 22:49:41.351910 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333252    4301 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 22:49:41.351910 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333254    4301 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 22:49:41.351910 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333257    4301 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 22:49:41.351910 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333260    4301 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 22:49:41.351910 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333262    4301 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 22:49:41.351910 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333265    4301 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 22:49:41.351910 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333267    4301 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 22:49:41.351910 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333270    4301 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 22:49:41.351910 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333272    4301 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 22:49:41.351910 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333275    4301 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 22:49:41.351910 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333278    4301 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 22:49:41.351910 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333280    4301 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 22:49:41.351910 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333285    4301 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 22:49:41.351910 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333287    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 22:49:41.352817 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333291    4301 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 22:49:41.352817 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333295    4301 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 22:49:41.352817 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333299    4301 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 22:49:41.352817 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333302    4301 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 22:49:41.352817 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333305    4301 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 22:49:41.352817 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333308    4301 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 22:49:41.352817 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333310    4301 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 22:49:41.352817 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333313    4301 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 22:49:41.352817 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333315    4301 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 22:49:41.352817 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333318    4301 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 22:49:41.352817 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333321    4301 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 22:49:41.352817 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333323    4301 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 22:49:41.352817 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333326    4301 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 22:49:41.352817 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333328    4301 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 22:49:41.352817 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333331    4301 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 22:49:41.352817 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333333    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 22:49:41.352817 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333336    4301 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 22:49:41.352817 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333338    4301 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 22:49:41.352817 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333341    4301 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 22:49:41.353656 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333344    4301 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 22:49:41.353656 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333350    4301 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 22:49:41.353656 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333352    4301 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 22:49:41.353656 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333368    4301 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 22:49:41.353656 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333372    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 22:49:41.353656 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333375    4301 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 22:49:41.353656 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333377    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 22:49:41.353656 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333380    4301 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 22:49:41.353656 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333383    4301 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 22:49:41.353656 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333386    4301 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 22:49:41.353656 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333389    4301 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 22:49:41.353656 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333393    4301 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 22:49:41.353656 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333396    4301 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 22:49:41.353656 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333398    4301 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 22:49:41.353656 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333401    4301 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 22:49:41.353656 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333406    4301 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 22:49:41.353656 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333410    4301 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 22:49:41.353656 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333413    4301 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 22:49:41.353656 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333416    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 22:49:41.354438 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333419    4301 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 22:49:41.354438 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333422    4301 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 22:49:41.354438 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333425    4301 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 22:49:41.354438 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333427    4301 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 22:49:41.354438 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333430    4301 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 22:49:41.354438 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333432    4301 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 22:49:41.354438 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333435    4301 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 22:49:41.354438 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:41.333438    4301 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 22:49:41.354438 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.333494    4301 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 22:49:41.354438 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.334499    4301 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 22:49:41.354438 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.334569    4301 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 22:49:41.354438 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.334750    4301 server.go:1014] "Starting client certificate rotation"
Oct 31 22:49:41.354438 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.334885    4301 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 22:49:41.354438 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.334927    4301 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 22:44:01 +0000 UTC" deadline="2028-04-12 11:40:35.706520676 +0000 UTC"
Oct 31 22:49:41.354438 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.334954    4301 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12684h50m54.371568777s"
Oct 31 22:49:41.354438 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.335228    4301 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 22:49:41.355080 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.336353    4301 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 22:49:41.355080 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.338061    4301 log.go:25] "Validated CRI v1 runtime API"
Oct 31 22:49:41.355080 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.339992    4301 log.go:25] "Validated CRI v1 image API"
Oct 31 22:49:41.355080 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.340982    4301 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 22:49:41.355080 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.342819    4301 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 aab09ba8-6576-46e8-af39-036eb8136f00:/dev/nvme0n1p4 f9f5192e-056d-4fe0-9a3c-378434ef4a61:/dev/nvme0n1p3]
Oct 31 22:49:41.355224 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.342841    4301 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/06263e2bcd2200a9857b0e6aa89953bc1872c669e9ed1bd071d6c7dac6538c17/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/06263e2bcd2200a9857b0e6aa89953bc1872c669e9ed1bd071d6c7dac6538c17/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e561ae6e12ededa8b6b6e6a3fce137075725798d98541edaa6fe26672b64795/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e561ae6e12ededa8b6b6e6a3fce137075725798d98541edaa6fe26672b64795/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4150b4496c8eefc7e46cde1c0ecf2e8357f6f408e516de83d24896e615ad921d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4150b4496c8eefc7e46cde1c0ecf2e8357f6f408e516de83d24896e615ad921d/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/55ff80fb1f1ede089ba94e10bdda200ea041a773ca473499ea65aa9eb5a606d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/55ff80fb1f1ede089ba94e10bdda200ea041a773ca473499ea65aa9eb5a606d3/userdata/shm major:0 minor:201 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ce77a3a926da5451f810783d142a98705e9ab2da818db1993298337df4d1dd1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ce77a3a926da5451f810783d142a98705e9ab2da818db1993298337df4d1dd1/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a084ce34ca6837ff5bc61c82598ceaa97d2b8a06e6f7f12be6d8c19674a7bd34/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a084ce34ca6837ff5bc61c82598ceaa97d2b8a06e6f7f12be6d8c19674a7bd34/userdata/shm major:0 minor:100 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ae9de0cb7f6fcc6e803269ca80675b8b420540a634f833c1b4d0c4d16f8d8d13/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ae9de0cb7f6fcc6e803269ca80675b8b420540a634f833c1b4d0c4d16f8d8d13/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b78e12a84694fd87b17b1f10e7e504fd2f790910e0790be13a308a12db41ea25/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b78e12a84694fd87b17b1f10e7e504fd2f790910e0790be13a308a12db41ea25/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/deb97a8d07b24b22d886a5dd7163e7f7136f13953737e321fb426eff2de3c50e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/deb97a8d07b24b22d886a5dd7163e7f7136f13953737e321fb426eff2de3c50e/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ea43db9120d0af582aeffe0e6a23e113d34310df2e662c0254186be7370b081d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ea43db9120d0af582aeffe0e6a23e113d34310df2e662c0254186be7370b081d/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed769345b7c93c550724a838b4f5c16a2010991acc92be55b05b2cf30a770075/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed769345b7c93c550724a838b4f5c16a2010991acc92be55b05b2cf30a770075/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa7a63b33e9aec9debd129fc805777e20079a37d22133f1983458413291498d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa7a63b33e9aec9debd129fc805777e20079a37d22133f1983458413291498d9/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f0042ee-6258-4aa8-af48-7568f2f2e8a1/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/1f0042ee-6258-4aa8-af48-7568f2f2e8a1/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f0042ee-6258-4aa8-af48-7568f2f2e8a1/volumes/kubernetes.io~projected/kube-api-access-r4sx8:{mountpoint:/var/lib/kubelet/pods/1f0042ee-6258-4aa8-af48-7568f2f2e8a1/volumes/kubernetes.io~projected/kube-api-access-r4sx8 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f0042ee-6258-4aa8-af48-7568f2f2e8a1/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/1f0042ee-6258-4aa8-af48-7568f2f2e8a1/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/546c0de6-9661-41cf-a068-cbbf88d05c2e/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/546c0de6-9661-41cf-a068-cbbf88d05c2e/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/546c0de6-9661-41cf-a068-cbbf88d05c2e/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/546c0de6-9661-41cf-a068-cbbf88d05c2e/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/546c0de6-9661-41cf-a068-cbbf88d05c2e/volumes/kubernetes.io~projected/kube-api-access-4bz55:{mountpoint:/var/lib/kubelet/pods/546c0de6-9661-41cf-a068-cbbf88d05c2e/volumes/kubernetes.io~projected/kube-api-access-4bz55 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f102a31-27f2-410b-9e36-2cdb8d9e852c/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/5f102a31-27f2-410b-9e36-2cdb8d9e852c/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/620515c1-daf9-48e9-9718-a71057b3b615/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/620515c1-daf9-48e9-9718-a71057b3b615/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65821d7f-a625-4696-9908-44a8eb6ecdde/volumes/kubernetes.io~projected/kube-api-access-mz9d9:{mountpoint:/var/lib/kubelet/pods/65821d7f-a625-4696-9908-44a8eb6ecdde/volumes/kubernetes.io~projected/kube-api-access-mz9d9 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76488729-53ad-4536-8157-7d41ef49863b/volumes/kubernetes.io~projected/kube-api-access-9rwp2:{mountpoint:/var/lib/kubelet/pods/76488729-53ad-4536-8157-7d41ef49863b/volumes/kubernetes.io~projected/kube-api-access-9rwp2 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81cc6a3e-e646-4287-84a7-d6830c40784b/volumes/kubernetes.io~projected/kube-api-access-7cxsk:{mountpoint:/var/lib/kubelet/pods/81cc6a3e-e646-4287-84a7-d6830c40784b/volumes/kubernetes.io~projected/kube-api-access-7cxsk major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b1db31b-aec0-495b-8b0c-4abbde3acb28/volumes/kubernetes.io~projected/kube-api-access-hdr9n:{mountpoint:/var/lib/kubelet/pods/8b1db31b-aec0-495b-8b0c-4abbde3acb28/volumes/kubernetes.io~projected/kube-api-access-hdr9n major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2ea746c-d257-4b00-be3e-ba726bd549de/volumes/kubernetes.io~projected/kube-api-access-spqnp:{mountpoint:/var/lib/kubelet/pods/a2ea746c-d257-4b00-be3e-ba726bd549de/volumes/kubernetes.io~projected/kube-api-access-spqnp major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16/volumes/kubernetes.io~projected/kube-api-access-79xfm:{mountpoint:/var/lib/kubelet/pods/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16/volumes/kubernetes.io~projected/kube-api-access-79xfm major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e46ea70b-37a4-4934-920c-ecdab212ecc6/volumes/kubernetes.io~projected/kube-api-access-5qndh:{mountpoint:/var/lib/kubelet/pods/e46ea70b-37a4-4934-920c-ecdab212ecc6/volumes/kubernetes.io~projected/kube-api-access-5qndh major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/efec9075-8e74-475d-8160-48e356140743/volumes/kubernetes.io~projected/kube-api-access-9hqz8:{mountpoint:/var/lib/kubelet/pods/efec9075-8e74-475d-8160-48e356140743/volumes/kubernetes.io~projected/kube-api-access-9hqz8 major:0 minor:197 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/fad60c6d03f5ec7b02688c70862ac15734357d7b3a39e0d01807f496b014294b/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/f8f3b21b4159027a7da72d189090ddfc9742849c2a658b4a44c1b09dd7503776/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/882ab6854877d568ad9b7cb06dff9808e5836525bdb8a31912bf68e82d64893d/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/ff9f4cd3a3fde506c16bb8973cb19c3e25734f19df85bdefe4c831079224388e/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/6e8ed007cd7a10364a422cf7f23651a4d5a0c381699913448241e9992528f0ac/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/35ca5232f121ec1727115b183d871c09ee467b158bd71db38d38c8ef368ab4aa/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/25bdc3e1b7eb9ecf5e01856b0880fd9ed583cbb17bcfce894b8e3c1c0fba6f7e/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/4e64cbcc27aa1d22b28fc4502e0e603d3ebed3e5e37f34b1300a5ed2816e62fc/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/caf3042b7e1a08e21184868eb2c389d711f1b7ce828a8c3997295b2928955958/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/0f2d6ca273d62babb12139b8d5ba2a6b8108c3dfb78ee7f29284d1431408bb70/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/973ef6faa9de8fcb9dd706a4883f3cd2e8dd9971edde0571b3afba81c62f6437/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/04846b335e401ffec03d2bc0e2b42e96af3d1ece0ea3860c1739c87bcfa0cb11/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/9fa4c302689df5b17d9cf624b5461eabc798b63a08116ff102427dacf42043d3/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/ea7b0a77f36383e4985f36896f80549170c22b11d37480fb364aca7ec6b904ac/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-171:{mountpoint:/var/lib/containers/storage/overlay/2806211a6499ac6e73b000a0fa72b0b7d6f0b50671fdd97e25fa40d59317f6a9/merged major:0 minor:171 fsType:overlay blockSize:0} overlay_0-173:{mountpoint:/var/lib/containers/storage/overlay/d1fbe7d4788e91eef2760a65f3fd1e2938dc97f7fce414693608e3d49dd4c95c/merged major:0 minor:173 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/20c67660afda2d7af5bc14779d0102d07c4c0df0c5c3f0185914e37981599d1a/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/6ab1d1db5cfbb65f39411517ad9d36c46ea5ff45d21b491c5840b5f98ec5447e/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/c5e148e0f6f9483a80d1aa4725e5cc920e9843265b69cd0714cdf2d7dbbd099a/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/6b8f1e5896aecbd5fa13ca634d29fa3001ff549bfe22120b12d9f4214f46b73c/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/56b89f9eabb036948e540bec1044650d3d1773e83bf05a2c7bdb09328914e2ba/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/f89e34637bbfb27af515719095b67f186089c1b39a88f37f2696edfc885ba879/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/30e80ffc50aeecd9bc94e387ebb50781e070b52b76092131bbcc39dd3c738a01/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/ca66905ba29ace2c3e9fcbccc1b720d6ff7a5aef0561f9c336953d7801879be6/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/d61e1047dc5144dee9ffc25bcc07a89050a363b9677c6663a3ef4611c8bcf15d/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/78ffcda643a74d3356d46f4cc9500720df98724b201987270be4ee84e2e13da3/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/6bda963a3863ce880ce5395bfe484ba942f0f3bf31c65187e0d88d3e3e768475/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/8e86060d1d436c8736f74d04a7196b1a4bdac14455a632096d99905934416239/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/5bc7597947c1f845f016b8dce7e6276091a178109048f35d62ac8d4a15482e35/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/db1d6003b8ecbaa0223b20320f1ce6be31968d513588553e8300bca6b24706b5/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/df6a90ac4aabc1849093b523e167d7b9558a3cf4692fe4722253ba389cf0c7f8/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-87:{mountpoint:/var/lib/containers/storage/overlay/8b15c7e14b90a38b59b02ead0b79de4a560ad7c25c80b84c7de53b81a79e246f/merged major:0 minor:87 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/2657c40c0d50d82cec4bee65fe7469f1dee8f7fdb8544ae47a6e1a6ec33c3a2d/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/54c3f435854b8b840fc1c13e54d95707753b4daa8ef0f72e9972d532d7981665/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/ff2a27161d12dd709077451595585e44bb3eccba98769c0591761212c412733e/merged major:0 minor:98 fsType:overlay blockSize:0}]
Oct 31 22:49:41.355224 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.349166    4301 manager.go:217] Machine: {Timestamp:2026-10-31 22:49:41.348665783 +0000 UTC m=+0.095855281 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3126893 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27926e91027f103fd8d11211f42a5f SystemUUID:ec27926e-9102-7f10-3fd8-d11211f42a5f BootID:79095032-3b68-469b-9072-952d60400029 Filesystems:[{Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/546c0de6-9661-41cf-a068-cbbf88d05c2e/volumes/kubernetes.io~projected/kube-api-access-4bz55 DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16/volumes/kubernetes.io~projected/kube-api-access-79xfm DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-87 DeviceMajor:0 DeviceMinor:87 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/8b1db31b-aec0-495b-8b0c-4abbde3acb28/volumes/kubernetes.io~projected/kube-api-access-hdr9n DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/546c0de6-9661-41cf-a068-cbbf88d05c2e/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1f0042ee-6258-4aa8-af48-7568f2f2e8a1/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/620515c1-daf9-48e9-9718-a71057b3b615/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/65821d7f-a625-4696-9908-44a8eb6ecdde/volumes/kubernetes.io~projected/kube-api-access-mz9d9 DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/81cc6a3e-e646-4287-84a7-d6830c40784b/volumes/kubernetes.io~projected/kube-api-access-7cxsk DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1f0042ee-6258-4aa8-af48-7568f2f2e8a1/volumes/kubernetes.io~projected/kube-api-access-r4sx8 DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/546c0de6-9661-41cf-a068-cbbf88d05c2e/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/55ff80fb1f1ede089ba94e10bdda200ea041a773ca473499ea65aa9eb5a606d3/userdata/shm DeviceMajor:0 DeviceMinor:201 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5f102a31-27f2-410b-9e36-2cdb8d9e852c/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/deb97a8d07b24b22d886a5dd7163e7f7136f13953737e321fb426eff2de3c50e/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ae9de0cb7f6fcc6e803269ca80675b8b420540a634f833c1b4d0c4d16f8d8d13/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e46ea70b-37a4-4934-920c-ecdab212ecc6/volumes/kubernetes.io~projected/kube-api-access-5qndh DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ea43db9120d0af582aeffe0e6a23e113d34310df2e662c0254186be7370b081d/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed769345b7c93c550724a838b4f5c16a2010991acc92be55b05b2cf30a770075/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1f0042ee-6258-4aa8-af48-7568f2f2e8a1/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa7a63b33e9aec9debd129fc805777e20079a37d22133f1983458413291498d9/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-173 DeviceMajor:0 DeviceMinor:173 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/76488729-53ad-4536-8157-7d41ef49863b/volumes/kubernetes.io~projected/kube-api-access-9rwp2 DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a084ce34ca6837ff5bc61c82598ceaa97d2b8a06e6f7f12be6d8c19674a7bd34/userdata/shm DeviceMajor:0 DeviceMinor:100 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4150b4496c8eefc7e46cde1c0ecf2e8357f6f408e516de83d24896e615ad921d/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/efec9075-8e74-475d-8160-48e356140743/volumes/kubernetes.io~projected/kube-api-access-9hqz8 DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-171 DeviceMajor:0 DeviceMinor:171 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/06263e2bcd2200a9857b0e6aa89953bc1872c669e9ed1bd071d6c7dac6538c17/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a2ea746c-d257-4b00-be3e-ba726bd549de/volumes/kubernetes.io~projected/kube-api-access-spqnp DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e561ae6e12ededa8b6b6e6a3fce137075725798d98541edaa6fe26672b64795/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ce77a3a926da5451f810783d142a98705e9ab2da818db1993298337df4d1dd1/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b78e12a84694fd87b17b1f10e7e504fd2f790910e0790be13a308a12db41ea25/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:55ff80fb1f1ede0 MacAddress:b2:3d:e4:a6:4e:d4 Speed:10000 Mtu:8901} {Name:ae9de0cb7f6fcc6 MacAddress:b6:02:5b:6a:5f:08 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:da:ca:ad:07 Speed:0 Mtu:9001} {Name:br-int MacAddress:da:e9:1b:d6:2b:55 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:da:ca:ad:07 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:1a:54:75:0d:f5:4a Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ea:a8:2f:e8:99:b5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 22:49:41.355224 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.349415    4301 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 22:49:41.355224 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.349462    4301 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 22:49:41.355224 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.349794    4301 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 22:49:41.355224 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.349829    4301 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-163.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 22:49:41.355224 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.350034    4301 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 22:49:41.355224 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.350043    4301 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 22:49:41.355224 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.350058    4301 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 22:49:41.355224 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.350077    4301 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 22:49:41.355224 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.350170    4301 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 22:49:41.355224 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.350341    4301 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 22:49:41.355224 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.350456    4301 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 22:49:41.355224 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.350488    4301 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 22:49:41.355224 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.350498    4301 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 22:49:41.355772 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.350508    4301 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 22:49:41.355772 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.350516    4301 kubelet.go:405] "Adding apiserver pod source"
Oct 31 22:49:41.355772 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.350525    4301 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 22:49:41.355772 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.351111    4301 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 22:49:41.355772 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.351121    4301 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 22:49:41.355772 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.351688    4301 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 22:49:41.355772 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.353471    4301 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 22:49:41.355772 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.353481    4301 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 22:49:41.355772 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.353597    4301 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 22:49:41.355772 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.353797    4301 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 22:49:41.355772 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.353824    4301 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 22:49:41.355772 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.353922    4301 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 22:49:41.355772 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.353938    4301 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 22:49:41.355772 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.353945    4301 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 22:49:41.355772 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.353953    4301 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 22:49:41.355772 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.353961    4301 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 22:49:41.355772 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.353967    4301 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 22:49:41.355772 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.353978    4301 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 22:49:41.355772 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.353990    4301 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 22:49:41.355772 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.354000    4301 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 22:49:41.355772 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.354008    4301 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 22:49:41.355772 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.354017    4301 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 22:49:41.355772 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.354027    4301 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 22:49:41.355772 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.354064    4301 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 22:49:41.355772 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.354070    4301 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 22:49:41.357408 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.357383    4301 server.go:1263] "Started kubelet"
Oct 31 22:49:41.357471 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.357426    4301 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 22:49:41.358005 ip-10-0-135-163 systemd[1]: Started Kubernetes Kubelet.
Oct 31 22:49:41.358914 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.358854    4301 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 22:49:41.359565 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.357793    4301 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 22:49:41.359565 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.359163    4301 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 22:49:41.359565 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.359482    4301 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 22:49:41.363925 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.362121    4301 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 22:49:41.363925 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.362373    4301 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 22:49:41.363925 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.362512    4301 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 22:49:41.363925 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.362526    4301 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 22:49:41.363925 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.362625    4301 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 22:44:03 +0000 UTC" deadline="2028-08-02 20:07:14.715534418 +0000 UTC"
Oct 31 22:49:41.363925 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.362658    4301 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15381h17m33.352879636s"
Oct 31 22:49:41.363925 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.362663    4301 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 22:49:41.363925 ip-10-0-135-163 kubenswrapper[4301]: E1031 22:49:41.363385    4301 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-163.us-east-2.compute.internal\" not found"
Oct 31 22:49:41.364478 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.364130    4301 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f0042ee-6258-4aa8-af48-7568f2f2e8a1" volumeName="kubernetes.io/secret/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 22:49:41.364478 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.364164    4301 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="546c0de6-9661-41cf-a068-cbbf88d05c2e" volumeName="kubernetes.io/empty-dir/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-tuned" seLinuxMountContext=""
Oct 31 22:49:41.364478 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.364183    4301 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f102a31-27f2-410b-9e36-2cdb8d9e852c" volumeName="kubernetes.io/configmap/5f102a31-27f2-410b-9e36-2cdb8d9e852c-konnectivity-ca" seLinuxMountContext=""
Oct 31 22:49:41.364478 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.364197    4301 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65821d7f-a625-4696-9908-44a8eb6ecdde" volumeName="kubernetes.io/projected/65821d7f-a625-4696-9908-44a8eb6ecdde-kube-api-access-mz9d9" seLinuxMountContext=""
Oct 31 22:49:41.364478 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.364211    4301 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76488729-53ad-4536-8157-7d41ef49863b" volumeName="kubernetes.io/configmap/76488729-53ad-4536-8157-7d41ef49863b-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 22:49:41.364478 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.364228    4301 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76488729-53ad-4536-8157-7d41ef49863b" volumeName="kubernetes.io/configmap/76488729-53ad-4536-8157-7d41ef49863b-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 22:49:41.364478 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.364243    4301 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b1db31b-aec0-495b-8b0c-4abbde3acb28" volumeName="kubernetes.io/configmap/8b1db31b-aec0-495b-8b0c-4abbde3acb28-serviceca" seLinuxMountContext=""
Oct 31 22:49:41.364478 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.364259    4301 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16" volumeName="kubernetes.io/configmap/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-cni-binary-copy" seLinuxMountContext=""
Oct 31 22:49:41.364478 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.364276    4301 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f0042ee-6258-4aa8-af48-7568f2f2e8a1" volumeName="kubernetes.io/configmap/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 22:49:41.364478 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.364290    4301 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f0042ee-6258-4aa8-af48-7568f2f2e8a1" volumeName="kubernetes.io/projected/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-kube-api-access-r4sx8" seLinuxMountContext=""
Oct 31 22:49:41.364478 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.364306    4301 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="546c0de6-9661-41cf-a068-cbbf88d05c2e" volumeName="kubernetes.io/projected/546c0de6-9661-41cf-a068-cbbf88d05c2e-kube-api-access-4bz55" seLinuxMountContext=""
Oct 31 22:49:41.364478 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.364320    4301 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f102a31-27f2-410b-9e36-2cdb8d9e852c" volumeName="kubernetes.io/secret/5f102a31-27f2-410b-9e36-2cdb8d9e852c-agent-certs" seLinuxMountContext=""
Oct 31 22:49:41.364478 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.364349    4301 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65821d7f-a625-4696-9908-44a8eb6ecdde" volumeName="kubernetes.io/configmap/65821d7f-a625-4696-9908-44a8eb6ecdde-iptables-alerter-script" seLinuxMountContext=""
Oct 31 22:49:41.364478 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.364394    4301 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76488729-53ad-4536-8157-7d41ef49863b" volumeName="kubernetes.io/configmap/76488729-53ad-4536-8157-7d41ef49863b-cni-binary-copy" seLinuxMountContext=""
Oct 31 22:49:41.364478 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.364416    4301 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76488729-53ad-4536-8157-7d41ef49863b" volumeName="kubernetes.io/projected/76488729-53ad-4536-8157-7d41ef49863b-kube-api-access-9rwp2" seLinuxMountContext=""
Oct 31 22:49:41.364478 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.364428    4301 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81cc6a3e-e646-4287-84a7-d6830c40784b" volumeName="kubernetes.io/projected/81cc6a3e-e646-4287-84a7-d6830c40784b-kube-api-access-7cxsk" seLinuxMountContext=""
Oct 31 22:49:41.364478 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.364446    4301 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="546c0de6-9661-41cf-a068-cbbf88d05c2e" volumeName="kubernetes.io/empty-dir/546c0de6-9661-41cf-a068-cbbf88d05c2e-tmp" seLinuxMountContext=""
Oct 31 22:49:41.364478 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.364459    4301 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b1db31b-aec0-495b-8b0c-4abbde3acb28" volumeName="kubernetes.io/projected/8b1db31b-aec0-495b-8b0c-4abbde3acb28-kube-api-access-hdr9n" seLinuxMountContext=""
Oct 31 22:49:41.364478 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.364477    4301 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2ea746c-d257-4b00-be3e-ba726bd549de" volumeName="kubernetes.io/projected/a2ea746c-d257-4b00-be3e-ba726bd549de-kube-api-access-spqnp" seLinuxMountContext=""
Oct 31 22:49:41.365702 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.365627    4301 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16" volumeName="kubernetes.io/projected/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-kube-api-access-79xfm" seLinuxMountContext=""
Oct 31 22:49:41.365799 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.365757    4301 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e46ea70b-37a4-4934-920c-ecdab212ecc6" volumeName="kubernetes.io/projected/e46ea70b-37a4-4934-920c-ecdab212ecc6-kube-api-access-5qndh" seLinuxMountContext=""
Oct 31 22:49:41.365799 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.365772    4301 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="efec9075-8e74-475d-8160-48e356140743" volumeName="kubernetes.io/projected/efec9075-8e74-475d-8160-48e356140743-kube-api-access-9hqz8" seLinuxMountContext=""
Oct 31 22:49:41.365799 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.365787    4301 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f0042ee-6258-4aa8-af48-7568f2f2e8a1" volumeName="kubernetes.io/configmap/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-ovnkube-config" seLinuxMountContext=""
Oct 31 22:49:41.365931 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.365806    4301 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f0042ee-6258-4aa8-af48-7568f2f2e8a1" volumeName="kubernetes.io/configmap/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-env-overrides" seLinuxMountContext=""
Oct 31 22:49:41.365931 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.365822    4301 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="620515c1-daf9-48e9-9718-a71057b3b615" volumeName="kubernetes.io/secret/620515c1-daf9-48e9-9718-a71057b3b615-original-pull-secret" seLinuxMountContext=""
Oct 31 22:49:41.365931 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.365831    4301 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16" volumeName="kubernetes.io/configmap/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-multus-daemon-config" seLinuxMountContext=""
Oct 31 22:49:41.365931 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.365850    4301 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e46ea70b-37a4-4934-920c-ecdab212ecc6" volumeName="kubernetes.io/empty-dir/e46ea70b-37a4-4934-920c-ecdab212ecc6-tmp-dir" seLinuxMountContext=""
Oct 31 22:49:41.365931 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.365895    4301 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 22:49:41.365931 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.365905    4301 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 22:49:41.367531 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.367459    4301 factory.go:55] Registering systemd factory
Oct 31 22:49:41.367531 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.367480    4301 factory.go:223] Registration of the systemd container factory successfully
Oct 31 22:49:41.367746 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.367728    4301 factory.go:153] Registering CRI-O factory
Oct 31 22:49:41.367806 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.367749    4301 factory.go:223] Registration of the crio container factory successfully
Oct 31 22:49:41.367806 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.367802    4301 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 22:49:41.367904 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.367822    4301 factory.go:103] Registering Raw factory
Oct 31 22:49:41.367904 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.367840    4301 manager.go:1199] Started watching for new ooms in manager
Oct 31 22:49:41.368182 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.368168    4301 manager.go:319] Starting recovery of all containers
Oct 31 22:49:41.370413 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.370395    4301 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 22:49:41.370413 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.370391    4301 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 22:49:41.370572 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.370558    4301 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 22:49:41.371877 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.371848    4301 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 22:49:41.374405 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.374388    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s9nl_1f0042ee-6258-4aa8-af48-7568f2f2e8a1/ovn-acl-logging/0.log"
Oct 31 22:49:41.374853 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.374823    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s9nl_1f0042ee-6258-4aa8-af48-7568f2f2e8a1/ovn-acl-logging/0.log"
Oct 31 22:49:41.394545 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.394511    4301 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 22:49:41.395973 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.395951    4301 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 22:49:41.395973 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.395977    4301 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 22:49:41.396112 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.395994    4301 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 22:49:41.396112 ip-10-0-135-163 kubenswrapper[4301]: E1031 22:49:41.396032    4301 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 22:49:41.399596 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.399572    4301 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 22:49:41.411684 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.411654    4301 generic.go:356] "Generic (PLEG): container finished" podID="76488729-53ad-4536-8157-7d41ef49863b" containerID="9fcecf831bfc119e14455eb7a55efb32a3f759f5891de634ffe6b3a13b761256" exitCode=0
Oct 31 22:49:41.411684 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.411682    4301 generic.go:356] "Generic (PLEG): container finished" podID="76488729-53ad-4536-8157-7d41ef49863b" containerID="cdb3c230eaf4f87f9ba4cdcd9fb312d0c5c39dfa6f791c199f5c56ed4036357b" exitCode=0
Oct 31 22:49:41.411684 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.411689    4301 generic.go:356] "Generic (PLEG): container finished" podID="76488729-53ad-4536-8157-7d41ef49863b" containerID="5c8dbb094f04552530ef9c5a3e9b96095b84b36434931c518345fde7d662a65a" exitCode=0
Oct 31 22:49:41.411855 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.411697    4301 generic.go:356] "Generic (PLEG): container finished" podID="76488729-53ad-4536-8157-7d41ef49863b" containerID="1219019f7b6709dc5fdcfd50b093de9a96fdb6b48ad567f89c3fe0ef4953acd6" exitCode=0
Oct 31 22:49:41.411855 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.411702    4301 generic.go:356] "Generic (PLEG): container finished" podID="76488729-53ad-4536-8157-7d41ef49863b" containerID="e057954cc32c812adb180af8da989c99bce5340f07ac0bd22d1a21a8a25c3de7" exitCode=0
Oct 31 22:49:41.411855 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.411710    4301 generic.go:356] "Generic (PLEG): container finished" podID="76488729-53ad-4536-8157-7d41ef49863b" containerID="a5b8955efe0b96a7dd03bdb9a68acaec6c0734e0ce4f50fe2cef3fe92748cfce" exitCode=0
Oct 31 22:49:41.416191 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.416161    4301 generic.go:356] "Generic (PLEG): container finished" podID="8ec0c93a8e98312f4e27fee411631fef" containerID="a2ea395c9a9dd176750af59537229c9d91eff30fd976a82fd9ec502edc005c42" exitCode=0
Oct 31 22:49:41.422337 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.422318    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s9nl_1f0042ee-6258-4aa8-af48-7568f2f2e8a1/ovn-acl-logging/0.log"
Oct 31 22:49:41.422649 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.422630    4301 generic.go:356] "Generic (PLEG): container finished" podID="1f0042ee-6258-4aa8-af48-7568f2f2e8a1" containerID="a192bd72d8082efb5034f801ff5a814b6a35d698cff6318cc13c92fd3e132d07" exitCode=1
Oct 31 22:49:41.427194 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.427139    4301 manager.go:324] Recovery completed
Oct 31 22:49:41.440607 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.440586    4301 cpu_manager.go:226] "Starting" policy="none"
Oct 31 22:49:41.440607 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.440602    4301 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 22:49:41.440717 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.440622    4301 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 22:49:41.440813 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.440797    4301 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 22:49:41.440843 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.440815    4301 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 22:49:41.440927 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.440917    4301 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 22:49:41.440958 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.440927    4301 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 22:49:41.440958 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.440934    4301 policy_none.go:50] "Start"
Oct 31 22:49:41.440958 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.440947    4301 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 22:49:41.441042 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.440959    4301 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 22:49:41.441099 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.441090    4301 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 22:49:41.441129 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.441099    4301 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 22:49:41.441129 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.441105    4301 policy_none.go:44] "Start"
Oct 31 22:49:41.445030 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.444997    4301 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 22:49:41.447561 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.447546    4301 manager.go:341] "Starting Device Plugin manager"
Oct 31 22:49:41.448037 ip-10-0-135-163 kubenswrapper[4301]: E1031 22:49:41.447574    4301 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 22:49:41.448037 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.448011    4301 server.go:87] "Starting device plugin registration server"
Oct 31 22:49:41.448213 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.448200    4301 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 22:49:41.448274 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.448215    4301 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 22:49:41.448334 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.448314    4301 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 22:49:41.448484 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.448453    4301 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 22:49:41.448567 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.448496    4301 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 22:49:41.448567 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.448504    4301 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 22:49:41.448647 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.448543    4301 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T22:49:41.448478538Z","UUID":"30c97acf-cae7-499a-b045-8a8a40898661","Handler":null,"Name":"","Endpoint":""}
Oct 31 22:49:41.449563 ip-10-0-135-163 kubenswrapper[4301]: E1031 22:49:41.449540    4301 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 22:49:41.449835 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.449809    4301 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 22:49:41.449835 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.449831    4301 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 22:49:41.496846 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.496791    4301 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-163.us-east-2.compute.internal"]
Oct 31 22:49:41.497277 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.497227    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal" event={"ID":"8ec0c93a8e98312f4e27fee411631fef","Type":"ContainerStarted","Data":"ef7c36772239aa412e173bfb2f7b2567d71bb227a4367b1dbfa25382fc5fe93c"}
Oct 31 22:49:41.497277 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.497269    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal" event={"ID":"8ec0c93a8e98312f4e27fee411631fef","Type":"ContainerDied","Data":"a2ea395c9a9dd176750af59537229c9d91eff30fd976a82fd9ec502edc005c42"}
Oct 31 22:49:41.497496 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.497289    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal" event={"ID":"8ec0c93a8e98312f4e27fee411631fef","Type":"ContainerStarted","Data":"06263e2bcd2200a9857b0e6aa89953bc1872c669e9ed1bd071d6c7dac6538c17"}
Oct 31 22:49:41.497496 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.497338    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-163.us-east-2.compute.internal" event={"ID":"2a808dc7f3e00ce75e0f6c4e3f550fac","Type":"ContainerStarted","Data":"8a9b910b0f109fa203a299360f5e977a4ff3c231ffe2f1f4f339f15024163c9f"}
Oct 31 22:49:41.497496 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.497378    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-163.us-east-2.compute.internal" event={"ID":"2a808dc7f3e00ce75e0f6c4e3f550fac","Type":"ContainerStarted","Data":"fa7a63b33e9aec9debd129fc805777e20079a37d22133f1983458413291498d9"}
Oct 31 22:49:41.497660 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.497531    4301 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:41.497660 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.497541    4301 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:41.508897 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.508880    4301 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 22:49:41.508980 ip-10-0-135-163 kubenswrapper[4301]: E1031 22:49:41.508926    4301 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-163.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:41.509503 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.509489    4301 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 22:49:41.509537 ip-10-0-135-163 kubenswrapper[4301]: E1031 22:49:41.509523    4301 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:41.548620 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.548598    4301 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 22:49:41.549897 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.549880    4301 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-163.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 22:49:41.549983 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.549912    4301 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-163.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 22:49:41.549983 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.549926    4301 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-163.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 22:49:41.549983 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.549954    4301 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:41.561040 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.561023    4301 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:41.561115 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.561077    4301 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:41.571927 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.571905    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8ec0c93a8e98312f4e27fee411631fef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal\" (UID: \"8ec0c93a8e98312f4e27fee411631fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:41.572003 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.571933    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ec0c93a8e98312f4e27fee411631fef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal\" (UID: \"8ec0c93a8e98312f4e27fee411631fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:41.572003 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.571952    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2a808dc7f3e00ce75e0f6c4e3f550fac-config\") pod \"kube-apiserver-proxy-ip-10-0-135-163.us-east-2.compute.internal\" (UID: \"2a808dc7f3e00ce75e0f6c4e3f550fac\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:41.673031 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.672994    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2a808dc7f3e00ce75e0f6c4e3f550fac-config\") pod \"kube-apiserver-proxy-ip-10-0-135-163.us-east-2.compute.internal\" (UID: \"2a808dc7f3e00ce75e0f6c4e3f550fac\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:41.673031 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.673031    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8ec0c93a8e98312f4e27fee411631fef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal\" (UID: \"8ec0c93a8e98312f4e27fee411631fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:41.673200 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.673048    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ec0c93a8e98312f4e27fee411631fef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal\" (UID: \"8ec0c93a8e98312f4e27fee411631fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:41.673200 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.673102    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ec0c93a8e98312f4e27fee411631fef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal\" (UID: \"8ec0c93a8e98312f4e27fee411631fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:41.673200 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.673105    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2a808dc7f3e00ce75e0f6c4e3f550fac-config\") pod \"kube-apiserver-proxy-ip-10-0-135-163.us-east-2.compute.internal\" (UID: \"2a808dc7f3e00ce75e0f6c4e3f550fac\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:41.673200 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:41.673105    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8ec0c93a8e98312f4e27fee411631fef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal\" (UID: \"8ec0c93a8e98312f4e27fee411631fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:42.350808 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.350771    4301 apiserver.go:52] "Watching apiserver"
Oct 31 22:49:42.356415 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.356390    4301 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 22:49:42.356741 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.356719    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-5qkg4","openshift-network-operator/iptables-alerter-lnhbm","kube-system/global-pull-secret-syncer-qv88p","kube-system/kube-apiserver-proxy-ip-10-0-135-163.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-dcczv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-2s8p8","openshift-network-diagnostics/network-check-target-ddxkp","openshift-ovn-kubernetes/ovnkube-node-7s9nl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2","openshift-image-registry/node-ca-xw8kl","openshift-multus/multus-7pksb","kube-system/konnectivity-agent-vc2j7","openshift-dns/node-resolver-cn8jx"]
Oct 31 22:49:42.360931 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.360901    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 22:49:42.361050 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.360954    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 22:49:42.361192 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.361167    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-l6qjt\""
Oct 31 22:49:42.361377 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.361172    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 22:49:42.361468 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.361270    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 22:49:42.361578 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.361557    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 22:49:42.361638 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.361611    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 22:49:42.361705 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.361561    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:42.361779 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.361396    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 22:49:42.361856 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.361470    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 22:49:42.361915 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.361773    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-2gb2z\""
Oct 31 22:49:42.361964 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.361946    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gr9dz\""
Oct 31 22:49:42.362497 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.362476    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 22:49:42.362631 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.362614    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-m4zzn\""
Oct 31 22:49:42.362698 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.362672    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 22:49:42.362742 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.362718    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 22:49:42.362824 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.362810    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 22:49:42.362867 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.362843    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 22:49:42.362913 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.362881    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pn67g\""
Oct 31 22:49:42.363537 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.363515    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-ssqvr\""
Oct 31 22:49:42.363788 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.363763    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gdsw9\""
Oct 31 22:49:42.363788 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.363775    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 22:49:42.364513 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.364492    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 22:49:42.364608 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.364565    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 22:49:42.364608 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.364582    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 22:49:42.364732 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.364620    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kg7vs\""
Oct 31 22:49:42.364732 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.364672    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dkn4d\""
Oct 31 22:49:42.364835 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.364745    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 22:49:42.364835 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.364798    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 22:49:42.365607 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.365586    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 22:49:42.365607 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.365599    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 22:49:42.365717 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.365622    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 22:49:42.365777 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.365762    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 22:49:42.365809 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.365762    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 22:49:42.365809 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.365783    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 22:49:42.365879 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.365827    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 22:49:42.367349 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.367334    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 22:49:42.367726 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.367700    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zbrj5\""
Oct 31 22:49:42.367726 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.367723    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 22:49:42.367882 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.367729    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2drsv\""
Oct 31 22:49:42.367882 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.367701    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 22:49:42.367882 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.367753    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 22:49:42.367882 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.367776    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-j2f6b\""
Oct 31 22:49:42.367882 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.367811    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 22:49:42.376616 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.376597    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-run-ovn-kubernetes\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.376661 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.376627    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cxsk\" (UniqueName: \"kubernetes.io/projected/81cc6a3e-e646-4287-84a7-d6830c40784b-kube-api-access-7cxsk\") pod \"network-metrics-daemon-5qkg4\" (UID: \"81cc6a3e-e646-4287-84a7-d6830c40784b\") " pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:42.376661 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.376649    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-sysconfig\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.376749 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.376672    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-cnibin\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.376749 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.376705    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-hostroot\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.376814 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.376774    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-etc-kubernetes\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.376814 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.376805    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79xfm\" (UniqueName: \"kubernetes.io/projected/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-kube-api-access-79xfm\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.376886 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.376870    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/620515c1-daf9-48e9-9718-a71057b3b615-kubelet-config\") pod \"global-pull-secret-syncer-qv88p\" (UID: \"620515c1-daf9-48e9-9718-a71057b3b615\") " pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:42.376916 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.376904    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-log-socket\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.376970 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.376928    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-sysctl-d\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.376970 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.376952    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5f102a31-27f2-410b-9e36-2cdb8d9e852c-konnectivity-ca\") pod \"konnectivity-agent-vc2j7\" (UID: \"5f102a31-27f2-410b-9e36-2cdb8d9e852c\") " pod="kube-system/konnectivity-agent-vc2j7"
Oct 31 22:49:42.377051 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.376985    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-etc-openvswitch\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.377092 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.377080    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/546c0de6-9661-41cf-a068-cbbf88d05c2e-tmp\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.377136 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.377097    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-device-dir\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:42.377170 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.377142    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-run-systemd\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.377213 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.377169    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-tuned\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.377213 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.377186    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rwp2\" (UniqueName: \"kubernetes.io/projected/76488729-53ad-4536-8157-7d41ef49863b-kube-api-access-9rwp2\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:42.377213 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.377188    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/546c0de6-9661-41cf-a068-cbbf88d05c2e-tmp\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.377346 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.377216    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65821d7f-a625-4696-9908-44a8eb6ecdde-host-slash\") pod \"iptables-alerter-lnhbm\" (UID: \"65821d7f-a625-4696-9908-44a8eb6ecdde\") " pod="openshift-network-operator/iptables-alerter-lnhbm"
Oct 31 22:49:42.377346 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.377250    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5f102a31-27f2-410b-9e36-2cdb8d9e852c-konnectivity-ca\") pod \"konnectivity-agent-vc2j7\" (UID: \"5f102a31-27f2-410b-9e36-2cdb8d9e852c\") " pod="kube-system/konnectivity-agent-vc2j7"
Oct 31 22:49:42.377346 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.377268    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-tuned\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.377346 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.377252    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-multus-conf-dir\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.377346 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.377325    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-ovnkube-config\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.377680 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.377378    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qndh\" (UniqueName: \"kubernetes.io/projected/e46ea70b-37a4-4934-920c-ecdab212ecc6-kube-api-access-5qndh\") pod \"node-resolver-cn8jx\" (UID: \"e46ea70b-37a4-4934-920c-ecdab212ecc6\") " pod="openshift-dns/node-resolver-cn8jx"
Oct 31 22:49:42.377680 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.377439    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76488729-53ad-4536-8157-7d41ef49863b-system-cni-dir\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:42.377680 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.377556    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-ovnkube-config\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.377680 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.377560    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/76488729-53ad-4536-8157-7d41ef49863b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:42.377680 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.377612    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz9d9\" (UniqueName: \"kubernetes.io/projected/65821d7f-a625-4696-9908-44a8eb6ecdde-kube-api-access-mz9d9\") pod \"iptables-alerter-lnhbm\" (UID: \"65821d7f-a625-4696-9908-44a8eb6ecdde\") " pod="openshift-network-operator/iptables-alerter-lnhbm"
Oct 31 22:49:42.377873 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.377680    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-multus-socket-dir-parent\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.377873 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.377704    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-run-netns\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.377873 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.377736    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-run-openvswitch\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.377873 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.377782    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-var-lib-kubelet\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.377873 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.377822    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-cni-binary-copy\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.377873 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.377857    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/76488729-53ad-4536-8157-7d41ef49863b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:42.377873 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.377862    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-multus-daemon-config\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.378095 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.377958    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5f102a31-27f2-410b-9e36-2cdb8d9e852c-agent-certs\") pod \"konnectivity-agent-vc2j7\" (UID: \"5f102a31-27f2-410b-9e36-2cdb8d9e852c\") " pod="kube-system/konnectivity-agent-vc2j7"
Oct 31 22:49:42.378095 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.378000    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-kubelet\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.378095 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.378027    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-run-netns\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.378095 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.378052    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-run-k8s-cni-cncf-io\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.378095 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.378060    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-cni-binary-copy\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.378330 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.378112    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-sys-fs\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:42.378330 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.378143    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/620515c1-daf9-48e9-9718-a71057b3b615-original-pull-secret\") pod \"global-pull-secret-syncer-qv88p\" (UID: \"620515c1-daf9-48e9-9718-a71057b3b615\") " pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:42.378330 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.378170    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-sysctl-conf\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.378330 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.378124    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-multus-daemon-config\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.378330 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.378193    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-systemd\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.378330 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.378284    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/76488729-53ad-4536-8157-7d41ef49863b-cnibin\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:42.378330 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.378304    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-node-log\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.378330 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.378296    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5f102a31-27f2-410b-9e36-2cdb8d9e852c-agent-certs\") pod \"konnectivity-agent-vc2j7\" (UID: \"5f102a31-27f2-410b-9e36-2cdb8d9e852c\") " pod="kube-system/konnectivity-agent-vc2j7"
Oct 31 22:49:42.378684 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.378323    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-ovn-node-metrics-cert\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.378684 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.378337    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/620515c1-daf9-48e9-9718-a71057b3b615-original-pull-secret\") pod \"global-pull-secret-syncer-qv88p\" (UID: \"620515c1-daf9-48e9-9718-a71057b3b615\") " pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:42.378684 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.378375    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-ovnkube-script-lib\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.378684 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.378402    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e46ea70b-37a4-4934-920c-ecdab212ecc6-hosts-file\") pod \"node-resolver-cn8jx\" (UID: \"e46ea70b-37a4-4934-920c-ecdab212ecc6\") " pod="openshift-dns/node-resolver-cn8jx"
Oct 31 22:49:42.378684 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.378441    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-kubernetes\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.378684 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.378530    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b1db31b-aec0-495b-8b0c-4abbde3acb28-host\") pod \"node-ca-xw8kl\" (UID: \"8b1db31b-aec0-495b-8b0c-4abbde3acb28\") " pod="openshift-image-registry/node-ca-xw8kl"
Oct 31 22:49:42.378684 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.378559    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-ovnkube-script-lib\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.378684 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.378560    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76488729-53ad-4536-8157-7d41ef49863b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:42.378684 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.378626    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/620515c1-daf9-48e9-9718-a71057b3b615-dbus\") pod \"global-pull-secret-syncer-qv88p\" (UID: \"620515c1-daf9-48e9-9718-a71057b3b615\") " pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:42.378684 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.378651    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-cni-netd\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.378684 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.378678    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4sx8\" (UniqueName: \"kubernetes.io/projected/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-kube-api-access-r4sx8\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.379110 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.378717    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-modprobe-d\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.379110 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.378784    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/76488729-53ad-4536-8157-7d41ef49863b-cni-binary-copy\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:42.379110 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.378811    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-ovn-node-metrics-cert\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.379110 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.378810    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-system-cni-dir\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.379110 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.378875    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-run-ovn\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.379110 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.378912    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-env-overrides\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.379110 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.378946    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-host\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.379110 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.378947    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/76488729-53ad-4536-8157-7d41ef49863b-cni-binary-copy\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:42.379110 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.378972    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8b1db31b-aec0-495b-8b0c-4abbde3acb28-serviceca\") pod \"node-ca-xw8kl\" (UID: \"8b1db31b-aec0-495b-8b0c-4abbde3acb28\") " pod="openshift-image-registry/node-ca-xw8kl"
Oct 31 22:49:42.379110 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.378990    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdr9n\" (UniqueName: \"kubernetes.io/projected/8b1db31b-aec0-495b-8b0c-4abbde3acb28-kube-api-access-hdr9n\") pod \"node-ca-xw8kl\" (UID: \"8b1db31b-aec0-495b-8b0c-4abbde3acb28\") " pod="openshift-image-registry/node-ca-xw8kl"
Oct 31 22:49:42.379110 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.379034    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-socket-dir\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:42.379110 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.379042    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-env-overrides\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.379441 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.379189    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-etc-selinux\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:42.379441 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.379227    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-run\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.379441 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.379264    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-multus-cni-dir\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.379441 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.379299    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spqnp\" (UniqueName: \"kubernetes.io/projected/a2ea746c-d257-4b00-be3e-ba726bd549de-kube-api-access-spqnp\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:42.379441 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.379327    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hqz8\" (UniqueName: \"kubernetes.io/projected/efec9075-8e74-475d-8160-48e356140743-kube-api-access-9hqz8\") pod \"network-check-target-ddxkp\" (UID: \"efec9075-8e74-475d-8160-48e356140743\") " pod="openshift-network-diagnostics/network-check-target-ddxkp"
Oct 31 22:49:42.379441 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.379343    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-systemd-units\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.379441 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.379374    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81cc6a3e-e646-4287-84a7-d6830c40784b-metrics-certs\") pod \"network-metrics-daemon-5qkg4\" (UID: \"81cc6a3e-e646-4287-84a7-d6830c40784b\") " pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:42.379641 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.379473    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bz55\" (UniqueName: \"kubernetes.io/projected/546c0de6-9661-41cf-a068-cbbf88d05c2e-kube-api-access-4bz55\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.379641 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.379504    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/76488729-53ad-4536-8157-7d41ef49863b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:42.379641 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.379530    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8b1db31b-aec0-495b-8b0c-4abbde3acb28-serviceca\") pod \"node-ca-xw8kl\" (UID: \"8b1db31b-aec0-495b-8b0c-4abbde3acb28\") " pod="openshift-image-registry/node-ca-xw8kl"
Oct 31 22:49:42.379641 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.379593    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-run-multus-certs\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.379749 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.379648    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-lib-modules\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.379749 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.379674    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-registration-dir\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:42.379749 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.379706    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.379749 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.379726    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-sys\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.379749 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.379745    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65821d7f-a625-4696-9908-44a8eb6ecdde-iptables-alerter-script\") pod \"iptables-alerter-lnhbm\" (UID: \"65821d7f-a625-4696-9908-44a8eb6ecdde\") " pod="openshift-network-operator/iptables-alerter-lnhbm"
Oct 31 22:49:42.379877 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.379742    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/76488729-53ad-4536-8157-7d41ef49863b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:42.379877 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.379805    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-var-lib-cni-bin\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.379877 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.379846    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-var-lib-cni-multus\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.379877 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.379865    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-var-lib-kubelet\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.380026 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.379880    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-slash\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.380026 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.379897    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-var-lib-openvswitch\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.380026 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.379912    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65821d7f-a625-4696-9908-44a8eb6ecdde-iptables-alerter-script\") pod \"iptables-alerter-lnhbm\" (UID: \"65821d7f-a625-4696-9908-44a8eb6ecdde\") " pod="openshift-network-operator/iptables-alerter-lnhbm"
Oct 31 22:49:42.380026 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.379913    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-cni-bin\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.380026 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.379938    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e46ea70b-37a4-4934-920c-ecdab212ecc6-tmp-dir\") pod \"node-resolver-cn8jx\" (UID: \"e46ea70b-37a4-4934-920c-ecdab212ecc6\") " pod="openshift-dns/node-resolver-cn8jx"
Oct 31 22:49:42.380026 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.380005    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:42.380193 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.380029    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e46ea70b-37a4-4934-920c-ecdab212ecc6-tmp-dir\") pod \"node-resolver-cn8jx\" (UID: \"e46ea70b-37a4-4934-920c-ecdab212ecc6\") " pod="openshift-dns/node-resolver-cn8jx"
Oct 31 22:49:42.386963 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.386945    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qndh\" (UniqueName: \"kubernetes.io/projected/e46ea70b-37a4-4934-920c-ecdab212ecc6-kube-api-access-5qndh\") pod \"node-resolver-cn8jx\" (UID: \"e46ea70b-37a4-4934-920c-ecdab212ecc6\") " pod="openshift-dns/node-resolver-cn8jx"
Oct 31 22:49:42.387039 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.386985    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rwp2\" (UniqueName: \"kubernetes.io/projected/76488729-53ad-4536-8157-7d41ef49863b-kube-api-access-9rwp2\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:42.387039 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.387015    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79xfm\" (UniqueName: \"kubernetes.io/projected/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-kube-api-access-79xfm\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.387224 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.387208    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cxsk\" (UniqueName: \"kubernetes.io/projected/81cc6a3e-e646-4287-84a7-d6830c40784b-kube-api-access-7cxsk\") pod \"network-metrics-daemon-5qkg4\" (UID: \"81cc6a3e-e646-4287-84a7-d6830c40784b\") " pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:42.387276 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.387225    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz9d9\" (UniqueName: \"kubernetes.io/projected/65821d7f-a625-4696-9908-44a8eb6ecdde-kube-api-access-mz9d9\") pod \"iptables-alerter-lnhbm\" (UID: \"65821d7f-a625-4696-9908-44a8eb6ecdde\") " pod="openshift-network-operator/iptables-alerter-lnhbm"
Oct 31 22:49:42.392789 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.392762    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hqz8\" (UniqueName: \"kubernetes.io/projected/efec9075-8e74-475d-8160-48e356140743-kube-api-access-9hqz8\") pod \"network-check-target-ddxkp\" (UID: \"efec9075-8e74-475d-8160-48e356140743\") " pod="openshift-network-diagnostics/network-check-target-ddxkp"
Oct 31 22:49:42.393008 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.392987    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdr9n\" (UniqueName: \"kubernetes.io/projected/8b1db31b-aec0-495b-8b0c-4abbde3acb28-kube-api-access-hdr9n\") pod \"node-ca-xw8kl\" (UID: \"8b1db31b-aec0-495b-8b0c-4abbde3acb28\") " pod="openshift-image-registry/node-ca-xw8kl"
Oct 31 22:49:42.393085 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.393030    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spqnp\" (UniqueName: \"kubernetes.io/projected/a2ea746c-d257-4b00-be3e-ba726bd549de-kube-api-access-spqnp\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:42.393143 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.393081    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4sx8\" (UniqueName: \"kubernetes.io/projected/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-kube-api-access-r4sx8\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.393143 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.393123    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bz55\" (UniqueName: \"kubernetes.io/projected/546c0de6-9661-41cf-a068-cbbf88d05c2e-kube-api-access-4bz55\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.429696 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.429678    4301 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:42.429859 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.429844    4301 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:42.440924 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.440905    4301 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 22:49:42.440998 ip-10-0-135-163 kubenswrapper[4301]: E1031 22:49:42.440942    4301 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:42.443797 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.443780    4301 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 22:49:42.443889 ip-10-0-135-163 kubenswrapper[4301]: E1031 22:49:42.443817    4301 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-163.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-163.us-east-2.compute.internal"
Oct 31 22:49:42.463029 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.463011    4301 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 22:49:42.480552 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.480530    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-sys-fs\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:42.480660 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.480557    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-sysctl-conf\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.480660 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.480572    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-systemd\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.480660 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.480587    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/76488729-53ad-4536-8157-7d41ef49863b-cnibin\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:42.480660 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.480601    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-node-log\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.480660 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.480641    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/76488729-53ad-4536-8157-7d41ef49863b-cnibin\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:42.480895 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.480662    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-sys-fs\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:42.480895 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.480699    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e46ea70b-37a4-4934-920c-ecdab212ecc6-hosts-file\") pod \"node-resolver-cn8jx\" (UID: \"e46ea70b-37a4-4934-920c-ecdab212ecc6\") " pod="openshift-dns/node-resolver-cn8jx"
Oct 31 22:49:42.480895 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.480714    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-node-log\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.480895 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.480730    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-kubernetes\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.480895 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.480751    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-sysctl-conf\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.480895 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.480765    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-systemd\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.480895 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.480772    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b1db31b-aec0-495b-8b0c-4abbde3acb28-host\") pod \"node-ca-xw8kl\" (UID: \"8b1db31b-aec0-495b-8b0c-4abbde3acb28\") " pod="openshift-image-registry/node-ca-xw8kl"
Oct 31 22:49:42.480895 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.480808    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b1db31b-aec0-495b-8b0c-4abbde3acb28-host\") pod \"node-ca-xw8kl\" (UID: \"8b1db31b-aec0-495b-8b0c-4abbde3acb28\") " pod="openshift-image-registry/node-ca-xw8kl"
Oct 31 22:49:42.480895 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.480809    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e46ea70b-37a4-4934-920c-ecdab212ecc6-hosts-file\") pod \"node-resolver-cn8jx\" (UID: \"e46ea70b-37a4-4934-920c-ecdab212ecc6\") " pod="openshift-dns/node-resolver-cn8jx"
Oct 31 22:49:42.480895 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.480814    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76488729-53ad-4536-8157-7d41ef49863b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:42.480895 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.480825    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-kubernetes\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.480895 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.480846    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/620515c1-daf9-48e9-9718-a71057b3b615-dbus\") pod \"global-pull-secret-syncer-qv88p\" (UID: \"620515c1-daf9-48e9-9718-a71057b3b615\") " pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:42.480895 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.480864    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76488729-53ad-4536-8157-7d41ef49863b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:42.480895 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.480867    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-cni-netd\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.481498 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.480904    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-modprobe-d\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.481498 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.480894    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-cni-netd\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.481498 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.480935    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-system-cni-dir\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.481498 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.480950    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/620515c1-daf9-48e9-9718-a71057b3b615-dbus\") pod \"global-pull-secret-syncer-qv88p\" (UID: \"620515c1-daf9-48e9-9718-a71057b3b615\") " pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:42.481498 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.480962    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-run-ovn\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.481498 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.480989    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-host\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.481498 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.481012    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-system-cni-dir\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.481498 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.481017    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-modprobe-d\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.481498 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.481013    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-socket-dir\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:42.481498 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.481032    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-run-ovn\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.481498 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.481057    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-host\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.481498 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.481061    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-etc-selinux\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:42.481498 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.481067    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-socket-dir\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:42.481498 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.481086    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-etc-selinux\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:42.481498 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.481088    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-run\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.481498 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.481113    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-multus-cni-dir\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.481498 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.481118    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-run\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.481498 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.481135    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-systemd-units\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.481498 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.481163    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-systemd-units\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.481498 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.481169    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81cc6a3e-e646-4287-84a7-d6830c40784b-metrics-certs\") pod \"network-metrics-daemon-5qkg4\" (UID: \"81cc6a3e-e646-4287-84a7-d6830c40784b\") " pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:42.481498 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.481190    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-multus-cni-dir\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.481498 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.481193    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-run-multus-certs\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.481498 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.481215    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-run-multus-certs\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.481498 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.481228    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-lib-modules\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.481498 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.481251    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-registration-dir\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:42.481498 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.481303    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-registration-dir\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:42.481498 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.481305    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.481498 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.481335    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-sys\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.481498 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.481346    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-lib-modules\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.482812 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.481626    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.482812 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.481656    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-var-lib-cni-bin\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.482812 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.481786    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-sys\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.482812 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.481811    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-var-lib-cni-bin\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.482812 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.481848    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-var-lib-cni-multus\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.482812 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.481909    4301 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 22:49:42.482812 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.481946    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-var-lib-kubelet\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.482812 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.482011    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-slash\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.482812 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.482092    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-var-lib-kubelet\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.482812 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.482128    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-var-lib-openvswitch\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.482812 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.482177    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-cni-bin\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.482812 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.482267    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:42.482812 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.482302    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-run-ovn-kubernetes\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.482812 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.482347    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-sysconfig\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.482812 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.482397    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-cnibin\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.482812 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.482434    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-hostroot\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.482812 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.482464    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-etc-kubernetes\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.482812 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.482586    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/620515c1-daf9-48e9-9718-a71057b3b615-kubelet-config\") pod \"global-pull-secret-syncer-qv88p\" (UID: \"620515c1-daf9-48e9-9718-a71057b3b615\") " pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:42.482812 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.482609    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-log-socket\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.482812 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.482640    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-sysctl-d\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.482812 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.482664    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:42.482812 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.482705    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-etc-openvswitch\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.482812 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.482716    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-slash\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.482812 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.482784    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-var-lib-openvswitch\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.483808 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.482834    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-run-ovn-kubernetes\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.483808 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.482902    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-cni-bin\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.483808 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.482902    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-cnibin\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.483808 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.482942    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-hostroot\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.483808 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.482954    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-sysconfig\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.483808 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.483008    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-etc-kubernetes\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.483808 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.483051    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-etc-openvswitch\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.483808 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.483085    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-device-dir\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:42.483808 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.483102    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-etc-sysctl-d\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.483808 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.483128    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-run-systemd\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.483808 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.483144    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-log-socket\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.483808 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.483200    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-run-systemd\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.483808 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.483192    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/620515c1-daf9-48e9-9718-a71057b3b615-kubelet-config\") pod \"global-pull-secret-syncer-qv88p\" (UID: \"620515c1-daf9-48e9-9718-a71057b3b615\") " pod="kube-system/global-pull-secret-syncer-qv88p"
Oct 31 22:49:42.483808 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.483269    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-var-lib-cni-multus\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.483808 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.483275    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a2ea746c-d257-4b00-be3e-ba726bd549de-device-dir\") pod \"aws-ebs-csi-driver-node-p48h2\" (UID: \"a2ea746c-d257-4b00-be3e-ba726bd549de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p48h2"
Oct 31 22:49:42.483808 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.483302    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65821d7f-a625-4696-9908-44a8eb6ecdde-host-slash\") pod \"iptables-alerter-lnhbm\" (UID: \"65821d7f-a625-4696-9908-44a8eb6ecdde\") " pod="openshift-network-operator/iptables-alerter-lnhbm"
Oct 31 22:49:42.483808 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.483338    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-multus-conf-dir\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.483808 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.483391    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65821d7f-a625-4696-9908-44a8eb6ecdde-host-slash\") pod \"iptables-alerter-lnhbm\" (UID: \"65821d7f-a625-4696-9908-44a8eb6ecdde\") " pod="openshift-network-operator/iptables-alerter-lnhbm"
Oct 31 22:49:42.483808 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.483404    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-multus-conf-dir\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.483808 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.483470    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76488729-53ad-4536-8157-7d41ef49863b-system-cni-dir\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:42.483808 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.483506    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-multus-socket-dir-parent\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.483808 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.483521    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76488729-53ad-4536-8157-7d41ef49863b-system-cni-dir\") pod \"multus-additional-cni-plugins-2s8p8\" (UID: \"76488729-53ad-4536-8157-7d41ef49863b\") " pod="openshift-multus/multus-additional-cni-plugins-2s8p8"
Oct 31 22:49:42.483808 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.483596    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-run-netns\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.483808 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.483626    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-run-netns\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.483808 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.483681    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-run-openvswitch\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.483808 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.483717    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-multus-socket-dir-parent\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.483808 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.483762    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-var-lib-kubelet\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.483808 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.483802    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-run-openvswitch\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.483808 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.483811    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/546c0de6-9661-41cf-a068-cbbf88d05c2e-var-lib-kubelet\") pod \"tuned-dcczv\" (UID: \"546c0de6-9661-41cf-a068-cbbf88d05c2e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcczv"
Oct 31 22:49:42.485032 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.483881    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-kubelet\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.485032 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.483924    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-kubelet\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.485032 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.483950    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-run-netns\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.485032 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.483994    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1f0042ee-6258-4aa8-af48-7568f2f2e8a1-host-run-netns\") pod \"ovnkube-node-7s9nl\" (UID: \"1f0042ee-6258-4aa8-af48-7568f2f2e8a1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:42.485032 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.484000    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-run-k8s-cni-cncf-io\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.485032 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.484071    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16-host-run-k8s-cni-cncf-io\") pod \"multus-7pksb\" (UID: \"a54fa4fc-fc3a-4fdc-9db4-3beded0c4e16\") " pod="openshift-multus/multus-7pksb"
Oct 31 22:49:42.485032 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.484634    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81cc6a3e-e646-4287-84a7-d6830c40784b-metrics-certs\") pod \"network-metrics-daemon-5qkg4\" (UID: \"81cc6a3e-e646-4287-84a7-d6830c40784b\") " pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:42.662406 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.662376    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qkg4"
Oct 31 22:49:42.957734 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.957660    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5qkg4"]
Oct 31 22:49:42.961870 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:42.961835    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81cc6a3e_e646_4287_84a7_d6830c40784b.slice/crio-39fd3c6021785994a487f504e65f4f7656a77a54ff9e7cbee542b888af4aad68 WatchSource:0}: Error finding container 39fd3c6021785994a487f504e65f4f7656a77a54ff9e7cbee542b888af4aad68: Status 404 returned error can't find the container with id 39fd3c6021785994a487f504e65f4f7656a77a54ff9e7cbee542b888af4aad68
Oct 31 22:49:42.964235 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.964213    4301 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 22:49:42.980544 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:42.980526    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ddxkp"
Oct 31 22:49:43.432716 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:43.432680    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5qkg4" event={"ID":"81cc6a3e-e646-4287-84a7-d6830c40784b","Type":"ContainerStarted","Data":"39fd3c6021785994a487f504e65f4f7656a77a54ff9e7cbee542b888af4aad68"}
Oct 31 22:49:44.436540 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:44.436504    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5qkg4" event={"ID":"81cc6a3e-e646-4287-84a7-d6830c40784b","Type":"ContainerStarted","Data":"96125ad6ef3ed0b2cfea0797a1543f8b7762bbdeb4aa8593ec901040a421ef78"}
Oct 31 22:49:44.436928 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:44.436555    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5qkg4" event={"ID":"81cc6a3e-e646-4287-84a7-d6830c40784b","Type":"ContainerStarted","Data":"ead62400a667c8d31fbef322e251ed94f3b7681defa25cec5267b50430491cad"}
Oct 31 22:49:45.350538 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:45.350504    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:45.978224 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:45.978195    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:45.978686 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:45.978409    4301 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 22:49:46.520700 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:46.520666    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7s9nl"
Oct 31 22:49:48.981318 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:48.981286    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vc2j7"
Oct 31 22:49:49.755525 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.755480    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-gcst4-bgj2j"]
Oct 31 22:49:49.757755 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.757734    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gcst4-bgj2j"
Oct 31 22:49:49.767807 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.767786    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-447ws\""
Oct 31 22:49:49.771556 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.770789    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5698547f59-qrxbm"]
Oct 31 22:49:49.771841 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.771820    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 22:49:49.771977 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.771924    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 22:49:49.771977 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.771952    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 22:49:49.772455 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.772439    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-784qh"]
Oct 31 22:49:49.772530 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.772516    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5698547f59-qrxbm"
Oct 31 22:49:49.774246 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.774218    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 22:49:49.774368 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.774305    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-784qh"
Oct 31 22:49:49.776564 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.776537    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-gcst4-bgj2j"]
Oct 31 22:49:49.778822 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.778800    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-n8vr2"]
Oct 31 22:49:49.780616 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.780598    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-t6v8n"]
Oct 31 22:49:49.780880 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.780778    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-n8vr2"
Oct 31 22:49:49.782176 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.782154    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t6v8n"
Oct 31 22:49:49.787173 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.787152    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-784qh"]
Oct 31 22:49:49.794537 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.794499    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5698547f59-qrxbm"]
Oct 31 22:49:49.800468 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.800445    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t6v8n"]
Oct 31 22:49:49.800586 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.800569    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-n8vr2"]
Oct 31 22:49:49.802820 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.802799    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 31 22:49:49.802908 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.802844    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 22:49:49.802908 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.802889    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 31 22:49:49.802973 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.802964    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 31 22:49:49.803040 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.803029    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 31 22:49:49.803075 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.803060    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 22:49:49.803106 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.803094    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 31 22:49:49.821104 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.821084    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mg6wn"]
Oct 31 22:49:49.823061 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.823043    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mg6wn"
Oct 31 22:49:49.823549 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.823520    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3625be12-bd2d-4134-b3a9-4f67638d0c4e-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-784qh\" (UID: \"3625be12-bd2d-4134-b3a9-4f67638d0c4e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-784qh"
Oct 31 22:49:49.823710 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.823693    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2425ea7b-ae01-49db-950b-8f5268a4c0d5-cert\") pod \"ingress-canary-t6v8n\" (UID: \"2425ea7b-ae01-49db-950b-8f5268a4c0d5\") " pod="openshift-ingress-canary/ingress-canary-t6v8n"
Oct 31 22:49:49.823820 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.823804    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/593497dc-c158-470b-9202-b81c33373e33-tmp\") pod \"console-5698547f59-qrxbm\" (UID: \"593497dc-c158-470b-9202-b81c33373e33\") " pod="openshift-console/console-5698547f59-qrxbm"
Oct 31 22:49:49.823950 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.823934    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt5fk\" (UniqueName: \"kubernetes.io/projected/ca495489-b79a-45ad-b566-68c7090b04c9-kube-api-access-zt5fk\") pod \"downloads-6dd789c8b7-n8vr2\" (UID: \"ca495489-b79a-45ad-b566-68c7090b04c9\") " pod="openshift-console/downloads-6dd789c8b7-n8vr2"
Oct 31 22:49:49.824049 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.824036    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0-archives-path\") pod \"periodic-gathering-gcst4-bgj2j\" (UID: \"a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0\") " pod="openshift-insights/periodic-gathering-gcst4-bgj2j"
Oct 31 22:49:49.824143 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.824131    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0-service-ca-bundle\") pod \"periodic-gathering-gcst4-bgj2j\" (UID: \"a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0\") " pod="openshift-insights/periodic-gathering-gcst4-bgj2j"
Oct 31 22:49:49.824264 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.824250    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0-trusted-ca-bundle\") pod \"periodic-gathering-gcst4-bgj2j\" (UID: \"a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0\") " pod="openshift-insights/periodic-gathering-gcst4-bgj2j"
Oct 31 22:49:49.824404 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.824391    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/593497dc-c158-470b-9202-b81c33373e33-console-serving-cert\") pod \"console-5698547f59-qrxbm\" (UID: \"593497dc-c158-470b-9202-b81c33373e33\") " pod="openshift-console/console-5698547f59-qrxbm"
Oct 31 22:49:49.824512 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.824500    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpxpd\" (UniqueName: \"kubernetes.io/projected/2425ea7b-ae01-49db-950b-8f5268a4c0d5-kube-api-access-cpxpd\") pod \"ingress-canary-t6v8n\" (UID: \"2425ea7b-ae01-49db-950b-8f5268a4c0d5\") " pod="openshift-ingress-canary/ingress-canary-t6v8n"
Oct 31 22:49:49.824806 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.824783    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/593497dc-c158-470b-9202-b81c33373e33-console-oauth-config\") pod \"console-5698547f59-qrxbm\" (UID: \"593497dc-c158-470b-9202-b81c33373e33\") " pod="openshift-console/console-5698547f59-qrxbm"
Oct 31 22:49:49.824874 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.824847    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/593497dc-c158-470b-9202-b81c33373e33-oauth-serving-cert\") pod \"console-5698547f59-qrxbm\" (UID: \"593497dc-c158-470b-9202-b81c33373e33\") " pod="openshift-console/console-5698547f59-qrxbm"
Oct 31 22:49:49.824960 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.824945    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca495489-b79a-45ad-b566-68c7090b04c9-tmp\") pod \"downloads-6dd789c8b7-n8vr2\" (UID: \"ca495489-b79a-45ad-b566-68c7090b04c9\") " pod="openshift-console/downloads-6dd789c8b7-n8vr2"
Oct 31 22:49:49.825012 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.824979    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/593497dc-c158-470b-9202-b81c33373e33-service-ca\") pod \"console-5698547f59-qrxbm\" (UID: \"593497dc-c158-470b-9202-b81c33373e33\") " pod="openshift-console/console-5698547f59-qrxbm"
Oct 31 22:49:49.825058 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.825013    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/593497dc-c158-470b-9202-b81c33373e33-console-config\") pod \"console-5698547f59-qrxbm\" (UID: \"593497dc-c158-470b-9202-b81c33373e33\") " pod="openshift-console/console-5698547f59-qrxbm"
Oct 31 22:49:49.825058 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.825045    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s6b8\" (UniqueName: \"kubernetes.io/projected/593497dc-c158-470b-9202-b81c33373e33-kube-api-access-6s6b8\") pod \"console-5698547f59-qrxbm\" (UID: \"593497dc-c158-470b-9202-b81c33373e33\") " pod="openshift-console/console-5698547f59-qrxbm"
Oct 31 22:49:49.825153 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.825078    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wq5r\" (UniqueName: \"kubernetes.io/projected/a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0-kube-api-access-7wq5r\") pod \"periodic-gathering-gcst4-bgj2j\" (UID: \"a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0\") " pod="openshift-insights/periodic-gathering-gcst4-bgj2j"
Oct 31 22:49:49.836854 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.836831    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mg6wn"]
Oct 31 22:49:49.856638 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.856619    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 22:49:49.873790 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.873767    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 22:49:49.874112 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.874089    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-5j2zg\""
Oct 31 22:49:49.874174 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.874102    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-cbxkt\""
Oct 31 22:49:49.874174 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.874102    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-s2g2v\""
Oct 31 22:49:49.874174 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.874134    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 31 22:49:49.874293 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.874184    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 22:49:49.874293 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.874216    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 31 22:49:49.874293 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.874250    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-845tz\""
Oct 31 22:49:49.874510 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.874303    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-n9xrd\""
Oct 31 22:49:49.875089 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.875073    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7qrj2"]
Oct 31 22:49:49.875468 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.875452    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 31 22:49:49.876800 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.876786    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7qrj2"
Oct 31 22:49:49.895401 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.895378    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79bdd8d85d-qbfpp"]
Oct 31 22:49:49.897221 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.897205    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79bdd8d85d-qbfpp"
Oct 31 22:49:49.897893 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.897877    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 22:49:49.900224 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.900203    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7qrj2"]
Oct 31 22:49:49.911195 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.911176    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 31 22:49:49.919118 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.918510    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79bdd8d85d-qbfpp"]
Oct 31 22:49:49.926292 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.926272    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca495489-b79a-45ad-b566-68c7090b04c9-tmp\") pod \"downloads-6dd789c8b7-n8vr2\" (UID: \"ca495489-b79a-45ad-b566-68c7090b04c9\") " pod="openshift-console/downloads-6dd789c8b7-n8vr2"
Oct 31 22:49:49.926403 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.926306    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zddzl\" (UniqueName: \"kubernetes.io/projected/2a85016a-a525-4512-a83f-b355b784b58e-kube-api-access-zddzl\") pod \"insights-runtime-extractor-mg6wn\" (UID: \"2a85016a-a525-4512-a83f-b355b784b58e\") " pod="openshift-insights/insights-runtime-extractor-mg6wn"
Oct 31 22:49:49.926403 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.926330    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dpmg\" (UniqueName: \"kubernetes.io/projected/13faa1a0-2c8b-4bb0-9531-10462aa1c3f4-kube-api-access-2dpmg\") pod \"dns-default-7qrj2\" (UID: \"13faa1a0-2c8b-4bb0-9531-10462aa1c3f4\") " pod="openshift-dns/dns-default-7qrj2"
Oct 31 22:49:49.926403 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.926367    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5d845527-5948-4edd-baab-2fe6d2bcba4a-installation-pull-secrets\") pod \"image-registry-79bdd8d85d-qbfpp\" (UID: \"5d845527-5948-4edd-baab-2fe6d2bcba4a\") " pod="openshift-image-registry/image-registry-79bdd8d85d-qbfpp"
Oct 31 22:49:49.926541 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.926421    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/593497dc-c158-470b-9202-b81c33373e33-console-config\") pod \"console-5698547f59-qrxbm\" (UID: \"593497dc-c158-470b-9202-b81c33373e33\") " pod="openshift-console/console-5698547f59-qrxbm"
Oct 31 22:49:49.926584 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.926548    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13faa1a0-2c8b-4bb0-9531-10462aa1c3f4-metrics-tls\") pod \"dns-default-7qrj2\" (UID: \"13faa1a0-2c8b-4bb0-9531-10462aa1c3f4\") " pod="openshift-dns/dns-default-7qrj2"
Oct 31 22:49:49.926650 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.926635    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wq5r\" (UniqueName: \"kubernetes.io/projected/a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0-kube-api-access-7wq5r\") pod \"periodic-gathering-gcst4-bgj2j\" (UID: \"a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0\") " pod="openshift-insights/periodic-gathering-gcst4-bgj2j"
Oct 31 22:49:49.926695 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.926664    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a85016a-a525-4512-a83f-b355b784b58e-crio-socket\") pod \"insights-runtime-extractor-mg6wn\" (UID: \"2a85016a-a525-4512-a83f-b355b784b58e\") " pod="openshift-insights/insights-runtime-extractor-mg6wn"
Oct 31 22:49:49.926735 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.926695    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a85016a-a525-4512-a83f-b355b784b58e-data-volume\") pod \"insights-runtime-extractor-mg6wn\" (UID: \"2a85016a-a525-4512-a83f-b355b784b58e\") " pod="openshift-insights/insights-runtime-extractor-mg6wn"
Oct 31 22:49:49.926735 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.926695    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca495489-b79a-45ad-b566-68c7090b04c9-tmp\") pod \"downloads-6dd789c8b7-n8vr2\" (UID: \"ca495489-b79a-45ad-b566-68c7090b04c9\") " pod="openshift-console/downloads-6dd789c8b7-n8vr2"
Oct 31 22:49:49.926735 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.926722    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2425ea7b-ae01-49db-950b-8f5268a4c0d5-cert\") pod \"ingress-canary-t6v8n\" (UID: \"2425ea7b-ae01-49db-950b-8f5268a4c0d5\") " pod="openshift-ingress-canary/ingress-canary-t6v8n"
Oct 31 22:49:49.926883 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.926749    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/593497dc-c158-470b-9202-b81c33373e33-tmp\") pod \"console-5698547f59-qrxbm\" (UID: \"593497dc-c158-470b-9202-b81c33373e33\") " pod="openshift-console/console-5698547f59-qrxbm"
Oct 31 22:49:49.926883 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.926796    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5d845527-5948-4edd-baab-2fe6d2bcba4a-trusted-ca\") pod \"image-registry-79bdd8d85d-qbfpp\" (UID: \"5d845527-5948-4edd-baab-2fe6d2bcba4a\") " pod="openshift-image-registry/image-registry-79bdd8d85d-qbfpp"
Oct 31 22:49:49.926883 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.926845    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0-trusted-ca-bundle\") pod \"periodic-gathering-gcst4-bgj2j\" (UID: \"a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0\") " pod="openshift-insights/periodic-gathering-gcst4-bgj2j"
Oct 31 22:49:49.926883 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.926871    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpxpd\" (UniqueName: \"kubernetes.io/projected/2425ea7b-ae01-49db-950b-8f5268a4c0d5-kube-api-access-cpxpd\") pod \"ingress-canary-t6v8n\" (UID: \"2425ea7b-ae01-49db-950b-8f5268a4c0d5\") " pod="openshift-ingress-canary/ingress-canary-t6v8n"
Oct 31 22:49:49.927025 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.926975    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt5fk\" (UniqueName: \"kubernetes.io/projected/ca495489-b79a-45ad-b566-68c7090b04c9-kube-api-access-zt5fk\") pod \"downloads-6dd789c8b7-n8vr2\" (UID: \"ca495489-b79a-45ad-b566-68c7090b04c9\") " pod="openshift-console/downloads-6dd789c8b7-n8vr2"
Oct 31 22:49:49.927025 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.927013    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/593497dc-c158-470b-9202-b81c33373e33-console-oauth-config\") pod \"console-5698547f59-qrxbm\" (UID: \"593497dc-c158-470b-9202-b81c33373e33\") " pod="openshift-console/console-5698547f59-qrxbm"
Oct 31 22:49:49.927117 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.927039    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/13faa1a0-2c8b-4bb0-9531-10462aa1c3f4-config-volume\") pod \"dns-default-7qrj2\" (UID: \"13faa1a0-2c8b-4bb0-9531-10462aa1c3f4\") " pod="openshift-dns/dns-default-7qrj2"
Oct 31 22:49:49.927117 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.927064    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtv47\" (UniqueName: \"kubernetes.io/projected/5d845527-5948-4edd-baab-2fe6d2bcba4a-kube-api-access-jtv47\") pod \"image-registry-79bdd8d85d-qbfpp\" (UID: \"5d845527-5948-4edd-baab-2fe6d2bcba4a\") " pod="openshift-image-registry/image-registry-79bdd8d85d-qbfpp"
Oct 31 22:49:49.927117 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.927110    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5d845527-5948-4edd-baab-2fe6d2bcba4a-registry-certificates\") pod \"image-registry-79bdd8d85d-qbfpp\" (UID: \"5d845527-5948-4edd-baab-2fe6d2bcba4a\") " pod="openshift-image-registry/image-registry-79bdd8d85d-qbfpp"
Oct 31 22:49:49.927255 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.927130    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/593497dc-c158-470b-9202-b81c33373e33-tmp\") pod \"console-5698547f59-qrxbm\" (UID: \"593497dc-c158-470b-9202-b81c33373e33\") " pod="openshift-console/console-5698547f59-qrxbm"
Oct 31 22:49:49.927255 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.927144    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/593497dc-c158-470b-9202-b81c33373e33-service-ca\") pod \"console-5698547f59-qrxbm\" (UID: \"593497dc-c158-470b-9202-b81c33373e33\") " pod="openshift-console/console-5698547f59-qrxbm"
Oct 31 22:49:49.927255 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.927149    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/593497dc-c158-470b-9202-b81c33373e33-console-config\") pod \"console-5698547f59-qrxbm\" (UID: \"593497dc-c158-470b-9202-b81c33373e33\") " pod="openshift-console/console-5698547f59-qrxbm"
Oct 31 22:49:49.927255 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.927212    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5d845527-5948-4edd-baab-2fe6d2bcba4a-image-registry-private-configuration\") pod \"image-registry-79bdd8d85d-qbfpp\" (UID: \"5d845527-5948-4edd-baab-2fe6d2bcba4a\") " pod="openshift-image-registry/image-registry-79bdd8d85d-qbfpp"
Oct 31 22:49:49.927472 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.927252    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s6b8\" (UniqueName: \"kubernetes.io/projected/593497dc-c158-470b-9202-b81c33373e33-kube-api-access-6s6b8\") pod \"console-5698547f59-qrxbm\" (UID: \"593497dc-c158-470b-9202-b81c33373e33\") " pod="openshift-console/console-5698547f59-qrxbm"
Oct 31 22:49:49.927472 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.927452    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3625be12-bd2d-4134-b3a9-4f67638d0c4e-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-784qh\" (UID: \"3625be12-bd2d-4134-b3a9-4f67638d0c4e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-784qh"
Oct 31 22:49:49.927576 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.927491    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a85016a-a525-4512-a83f-b355b784b58e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mg6wn\" (UID: \"2a85016a-a525-4512-a83f-b355b784b58e\") " pod="openshift-insights/insights-runtime-extractor-mg6wn"
Oct 31 22:49:49.927576 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.927518    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5d845527-5948-4edd-baab-2fe6d2bcba4a-ca-trust-extracted\") pod \"image-registry-79bdd8d85d-qbfpp\" (UID: \"5d845527-5948-4edd-baab-2fe6d2bcba4a\") " pod="openshift-image-registry/image-registry-79bdd8d85d-qbfpp"
Oct 31 22:49:49.927576 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.927544    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13faa1a0-2c8b-4bb0-9531-10462aa1c3f4-tmp-dir\") pod \"dns-default-7qrj2\" (UID: \"13faa1a0-2c8b-4bb0-9531-10462aa1c3f4\") " pod="openshift-dns/dns-default-7qrj2"
Oct 31 22:49:49.927576 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.927570    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0-archives-path\") pod \"periodic-gathering-gcst4-bgj2j\" (UID: \"a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0\") " pod="openshift-insights/periodic-gathering-gcst4-bgj2j"
Oct 31 22:49:49.927772 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.927593    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0-service-ca-bundle\") pod \"periodic-gathering-gcst4-bgj2j\" (UID: \"a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0\") " pod="openshift-insights/periodic-gathering-gcst4-bgj2j"
Oct 31 22:49:49.927772 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.927622    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a85016a-a525-4512-a83f-b355b784b58e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mg6wn\" (UID: \"2a85016a-a525-4512-a83f-b355b784b58e\") " pod="openshift-insights/insights-runtime-extractor-mg6wn"
Oct 31 22:49:49.927772 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.927648    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5d845527-5948-4edd-baab-2fe6d2bcba4a-registry-tls\") pod \"image-registry-79bdd8d85d-qbfpp\" (UID: \"5d845527-5948-4edd-baab-2fe6d2bcba4a\") " pod="openshift-image-registry/image-registry-79bdd8d85d-qbfpp"
Oct 31 22:49:49.927772 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.927672    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5d845527-5948-4edd-baab-2fe6d2bcba4a-bound-sa-token\") pod \"image-registry-79bdd8d85d-qbfpp\" (UID: \"5d845527-5948-4edd-baab-2fe6d2bcba4a\") " pod="openshift-image-registry/image-registry-79bdd8d85d-qbfpp"
Oct 31 22:49:49.927772 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.927710    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/593497dc-c158-470b-9202-b81c33373e33-console-serving-cert\") pod \"console-5698547f59-qrxbm\" (UID: \"593497dc-c158-470b-9202-b81c33373e33\") " pod="openshift-console/console-5698547f59-qrxbm"
Oct 31 22:49:49.927772 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.927739    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/593497dc-c158-470b-9202-b81c33373e33-oauth-serving-cert\") pod \"console-5698547f59-qrxbm\" (UID: \"593497dc-c158-470b-9202-b81c33373e33\") " pod="openshift-console/console-5698547f59-qrxbm"
Oct 31 22:49:49.928900 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.927913    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/593497dc-c158-470b-9202-b81c33373e33-service-ca\") pod \"console-5698547f59-qrxbm\" (UID: \"593497dc-c158-470b-9202-b81c33373e33\") " pod="openshift-console/console-5698547f59-qrxbm"
Oct 31 22:49:49.928900 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.928008    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0-trusted-ca-bundle\") pod \"periodic-gathering-gcst4-bgj2j\" (UID: \"a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0\") " pod="openshift-insights/periodic-gathering-gcst4-bgj2j"
Oct 31 22:49:49.928900 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.928053    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0-archives-path\") pod \"periodic-gathering-gcst4-bgj2j\" (UID: \"a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0\") " pod="openshift-insights/periodic-gathering-gcst4-bgj2j"
Oct 31 22:49:49.928900 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.928480    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/593497dc-c158-470b-9202-b81c33373e33-oauth-serving-cert\") pod \"console-5698547f59-qrxbm\" (UID: \"593497dc-c158-470b-9202-b81c33373e33\") " pod="openshift-console/console-5698547f59-qrxbm"
Oct 31 22:49:49.928900 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.928507    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0-service-ca-bundle\") pod \"periodic-gathering-gcst4-bgj2j\" (UID: \"a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0\") " pod="openshift-insights/periodic-gathering-gcst4-bgj2j"
Oct 31 22:49:49.929443 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.929424    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2425ea7b-ae01-49db-950b-8f5268a4c0d5-cert\") pod \"ingress-canary-t6v8n\" (UID: \"2425ea7b-ae01-49db-950b-8f5268a4c0d5\") " pod="openshift-ingress-canary/ingress-canary-t6v8n"
Oct 31 22:49:49.929534 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.929514    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/593497dc-c158-470b-9202-b81c33373e33-console-oauth-config\") pod \"console-5698547f59-qrxbm\" (UID: \"593497dc-c158-470b-9202-b81c33373e33\") " pod="openshift-console/console-5698547f59-qrxbm"
Oct 31 22:49:49.930085 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.930067    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/593497dc-c158-470b-9202-b81c33373e33-console-serving-cert\") pod \"console-5698547f59-qrxbm\" (UID: \"593497dc-c158-470b-9202-b81c33373e33\") " pod="openshift-console/console-5698547f59-qrxbm"
Oct 31 22:49:49.930498 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.930482    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3625be12-bd2d-4134-b3a9-4f67638d0c4e-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-784qh\" (UID: \"3625be12-bd2d-4134-b3a9-4f67638d0c4e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-784qh"
Oct 31 22:49:49.948805 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.948774    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt5fk\" (UniqueName: \"kubernetes.io/projected/ca495489-b79a-45ad-b566-68c7090b04c9-kube-api-access-zt5fk\") pod \"downloads-6dd789c8b7-n8vr2\" (UID: \"ca495489-b79a-45ad-b566-68c7090b04c9\") " pod="openshift-console/downloads-6dd789c8b7-n8vr2"
Oct 31 22:49:49.949002 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.948983    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpxpd\" (UniqueName: \"kubernetes.io/projected/2425ea7b-ae01-49db-950b-8f5268a4c0d5-kube-api-access-cpxpd\") pod \"ingress-canary-t6v8n\" (UID: \"2425ea7b-ae01-49db-950b-8f5268a4c0d5\") " pod="openshift-ingress-canary/ingress-canary-t6v8n"
Oct 31 22:49:49.949326 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.949307    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s6b8\" (UniqueName: \"kubernetes.io/projected/593497dc-c158-470b-9202-b81c33373e33-kube-api-access-6s6b8\") pod \"console-5698547f59-qrxbm\" (UID: \"593497dc-c158-470b-9202-b81c33373e33\") " pod="openshift-console/console-5698547f59-qrxbm"
Oct 31 22:49:49.949723 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.949707    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wq5r\" (UniqueName: \"kubernetes.io/projected/a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0-kube-api-access-7wq5r\") pod \"periodic-gathering-gcst4-bgj2j\" (UID: \"a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0\") " pod="openshift-insights/periodic-gathering-gcst4-bgj2j"
Oct 31 22:49:49.968131 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.968108    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 31 22:49:49.968468 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.968333    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 22:49:49.968468 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.968348    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 31 22:49:49.968468 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.968444    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hdfv2\""
Oct 31 22:49:49.968468 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.968453    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 31 22:49:49.968690 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:49.968274    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7g25v\""
Oct 31 22:49:50.029119 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.029030    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a85016a-a525-4512-a83f-b355b784b58e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mg6wn\" (UID: \"2a85016a-a525-4512-a83f-b355b784b58e\") " pod="openshift-insights/insights-runtime-extractor-mg6wn"
Oct 31 22:49:50.029119 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.029066    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5d845527-5948-4edd-baab-2fe6d2bcba4a-ca-trust-extracted\") pod \"image-registry-79bdd8d85d-qbfpp\" (UID: \"5d845527-5948-4edd-baab-2fe6d2bcba4a\") " pod="openshift-image-registry/image-registry-79bdd8d85d-qbfpp"
Oct 31 22:49:50.029119 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.029086    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13faa1a0-2c8b-4bb0-9531-10462aa1c3f4-tmp-dir\") pod \"dns-default-7qrj2\" (UID: \"13faa1a0-2c8b-4bb0-9531-10462aa1c3f4\") " pod="openshift-dns/dns-default-7qrj2"
Oct 31 22:49:50.029119 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.029106    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a85016a-a525-4512-a83f-b355b784b58e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mg6wn\" (UID: \"2a85016a-a525-4512-a83f-b355b784b58e\") " pod="openshift-insights/insights-runtime-extractor-mg6wn"
Oct 31 22:49:50.029782 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.029132    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5d845527-5948-4edd-baab-2fe6d2bcba4a-registry-tls\") pod \"image-registry-79bdd8d85d-qbfpp\" (UID: \"5d845527-5948-4edd-baab-2fe6d2bcba4a\") " pod="openshift-image-registry/image-registry-79bdd8d85d-qbfpp"
Oct 31 22:49:50.029782 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.029233    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5d845527-5948-4edd-baab-2fe6d2bcba4a-bound-sa-token\") pod \"image-registry-79bdd8d85d-qbfpp\" (UID: \"5d845527-5948-4edd-baab-2fe6d2bcba4a\") " pod="openshift-image-registry/image-registry-79bdd8d85d-qbfpp"
Oct 31 22:49:50.029782 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.029291    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zddzl\" (UniqueName: \"kubernetes.io/projected/2a85016a-a525-4512-a83f-b355b784b58e-kube-api-access-zddzl\") pod \"insights-runtime-extractor-mg6wn\" (UID: \"2a85016a-a525-4512-a83f-b355b784b58e\") " pod="openshift-insights/insights-runtime-extractor-mg6wn"
Oct 31 22:49:50.029782 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.029309    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dpmg\" (UniqueName: \"kubernetes.io/projected/13faa1a0-2c8b-4bb0-9531-10462aa1c3f4-kube-api-access-2dpmg\") pod \"dns-default-7qrj2\" (UID: \"13faa1a0-2c8b-4bb0-9531-10462aa1c3f4\") " pod="openshift-dns/dns-default-7qrj2"
Oct 31 22:49:50.029782 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.029329    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5d845527-5948-4edd-baab-2fe6d2bcba4a-installation-pull-secrets\") pod \"image-registry-79bdd8d85d-qbfpp\" (UID: \"5d845527-5948-4edd-baab-2fe6d2bcba4a\") " pod="openshift-image-registry/image-registry-79bdd8d85d-qbfpp"
Oct 31 22:49:50.029782 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.029379    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13faa1a0-2c8b-4bb0-9531-10462aa1c3f4-metrics-tls\") pod \"dns-default-7qrj2\" (UID: \"13faa1a0-2c8b-4bb0-9531-10462aa1c3f4\") " pod="openshift-dns/dns-default-7qrj2"
Oct 31 22:49:50.029782 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.029412    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a85016a-a525-4512-a83f-b355b784b58e-crio-socket\") pod \"insights-runtime-extractor-mg6wn\" (UID: \"2a85016a-a525-4512-a83f-b355b784b58e\") " pod="openshift-insights/insights-runtime-extractor-mg6wn"
Oct 31 22:49:50.029782 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.029442    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a85016a-a525-4512-a83f-b355b784b58e-data-volume\") pod \"insights-runtime-extractor-mg6wn\" (UID: \"2a85016a-a525-4512-a83f-b355b784b58e\") " pod="openshift-insights/insights-runtime-extractor-mg6wn"
Oct 31 22:49:50.029782 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.029470    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5d845527-5948-4edd-baab-2fe6d2bcba4a-trusted-ca\") pod \"image-registry-79bdd8d85d-qbfpp\" (UID: \"5d845527-5948-4edd-baab-2fe6d2bcba4a\") " pod="openshift-image-registry/image-registry-79bdd8d85d-qbfpp"
Oct 31 22:49:50.029782 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.029512    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/13faa1a0-2c8b-4bb0-9531-10462aa1c3f4-config-volume\") pod \"dns-default-7qrj2\" (UID: \"13faa1a0-2c8b-4bb0-9531-10462aa1c3f4\") " pod="openshift-dns/dns-default-7qrj2"
Oct 31 22:49:50.029782 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.029535    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtv47\" (UniqueName: \"kubernetes.io/projected/5d845527-5948-4edd-baab-2fe6d2bcba4a-kube-api-access-jtv47\") pod \"image-registry-79bdd8d85d-qbfpp\" (UID: \"5d845527-5948-4edd-baab-2fe6d2bcba4a\") " pod="openshift-image-registry/image-registry-79bdd8d85d-qbfpp"
Oct 31 22:49:50.029782 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.029538    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13faa1a0-2c8b-4bb0-9531-10462aa1c3f4-tmp-dir\") pod \"dns-default-7qrj2\" (UID: \"13faa1a0-2c8b-4bb0-9531-10462aa1c3f4\") " pod="openshift-dns/dns-default-7qrj2"
Oct 31 22:49:50.029782 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.029597    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5d845527-5948-4edd-baab-2fe6d2bcba4a-ca-trust-extracted\") pod \"image-registry-79bdd8d85d-qbfpp\" (UID: \"5d845527-5948-4edd-baab-2fe6d2bcba4a\") " pod="openshift-image-registry/image-registry-79bdd8d85d-qbfpp"
Oct 31 22:49:50.029782 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.029614    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5d845527-5948-4edd-baab-2fe6d2bcba4a-registry-certificates\") pod \"image-registry-79bdd8d85d-qbfpp\" (UID: \"5d845527-5948-4edd-baab-2fe6d2bcba4a\") " pod="openshift-image-registry/image-registry-79bdd8d85d-qbfpp"
Oct 31 22:49:50.029782 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.029675    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5d845527-5948-4edd-baab-2fe6d2bcba4a-image-registry-private-configuration\") pod \"image-registry-79bdd8d85d-qbfpp\" (UID: \"5d845527-5948-4edd-baab-2fe6d2bcba4a\") " pod="openshift-image-registry/image-registry-79bdd8d85d-qbfpp"
Oct 31 22:49:50.030508 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.029891    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a85016a-a525-4512-a83f-b355b784b58e-crio-socket\") pod \"insights-runtime-extractor-mg6wn\" (UID: \"2a85016a-a525-4512-a83f-b355b784b58e\") " pod="openshift-insights/insights-runtime-extractor-mg6wn"
Oct 31 22:49:50.030508 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.029904    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a85016a-a525-4512-a83f-b355b784b58e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mg6wn\" (UID: \"2a85016a-a525-4512-a83f-b355b784b58e\") " pod="openshift-insights/insights-runtime-extractor-mg6wn"
Oct 31 22:49:50.030508 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.030495    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5d845527-5948-4edd-baab-2fe6d2bcba4a-registry-certificates\") pod \"image-registry-79bdd8d85d-qbfpp\" (UID: \"5d845527-5948-4edd-baab-2fe6d2bcba4a\") " pod="openshift-image-registry/image-registry-79bdd8d85d-qbfpp"
Oct 31 22:49:50.030809 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.030787    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/13faa1a0-2c8b-4bb0-9531-10462aa1c3f4-config-volume\") pod \"dns-default-7qrj2\" (UID: \"13faa1a0-2c8b-4bb0-9531-10462aa1c3f4\") " pod="openshift-dns/dns-default-7qrj2"
Oct 31 22:49:50.030910 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.030869    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a85016a-a525-4512-a83f-b355b784b58e-data-volume\") pod \"insights-runtime-extractor-mg6wn\" (UID: \"2a85016a-a525-4512-a83f-b355b784b58e\") " pod="openshift-insights/insights-runtime-extractor-mg6wn"
Oct 31 22:49:50.030983 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.030942    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5d845527-5948-4edd-baab-2fe6d2bcba4a-trusted-ca\") pod \"image-registry-79bdd8d85d-qbfpp\" (UID: \"5d845527-5948-4edd-baab-2fe6d2bcba4a\") " pod="openshift-image-registry/image-registry-79bdd8d85d-qbfpp"
Oct 31 22:49:50.032373 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.032343    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5d845527-5948-4edd-baab-2fe6d2bcba4a-registry-tls\") pod \"image-registry-79bdd8d85d-qbfpp\" (UID: \"5d845527-5948-4edd-baab-2fe6d2bcba4a\") " pod="openshift-image-registry/image-registry-79bdd8d85d-qbfpp"
Oct 31 22:49:50.032496 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.032407    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13faa1a0-2c8b-4bb0-9531-10462aa1c3f4-metrics-tls\") pod \"dns-default-7qrj2\" (UID: \"13faa1a0-2c8b-4bb0-9531-10462aa1c3f4\") " pod="openshift-dns/dns-default-7qrj2"
Oct 31 22:49:50.032496 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.032420    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5d845527-5948-4edd-baab-2fe6d2bcba4a-installation-pull-secrets\") pod \"image-registry-79bdd8d85d-qbfpp\" (UID: \"5d845527-5948-4edd-baab-2fe6d2bcba4a\") " pod="openshift-image-registry/image-registry-79bdd8d85d-qbfpp"
Oct 31 22:49:50.032496 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.032450    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a85016a-a525-4512-a83f-b355b784b58e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mg6wn\" (UID: \"2a85016a-a525-4512-a83f-b355b784b58e\") " pod="openshift-insights/insights-runtime-extractor-mg6wn"
Oct 31 22:49:50.032496 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.032484    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5d845527-5948-4edd-baab-2fe6d2bcba4a-image-registry-private-configuration\") pod \"image-registry-79bdd8d85d-qbfpp\" (UID: \"5d845527-5948-4edd-baab-2fe6d2bcba4a\") " pod="openshift-image-registry/image-registry-79bdd8d85d-qbfpp"
Oct 31 22:49:50.040603 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.040584    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zddzl\" (UniqueName: \"kubernetes.io/projected/2a85016a-a525-4512-a83f-b355b784b58e-kube-api-access-zddzl\") pod \"insights-runtime-extractor-mg6wn\" (UID: \"2a85016a-a525-4512-a83f-b355b784b58e\") " pod="openshift-insights/insights-runtime-extractor-mg6wn"
Oct 31 22:49:50.043767 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.043744    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtv47\" (UniqueName: \"kubernetes.io/projected/5d845527-5948-4edd-baab-2fe6d2bcba4a-kube-api-access-jtv47\") pod \"image-registry-79bdd8d85d-qbfpp\" (UID: \"5d845527-5948-4edd-baab-2fe6d2bcba4a\") " pod="openshift-image-registry/image-registry-79bdd8d85d-qbfpp"
Oct 31 22:49:50.044116 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.044100    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dpmg\" (UniqueName: \"kubernetes.io/projected/13faa1a0-2c8b-4bb0-9531-10462aa1c3f4-kube-api-access-2dpmg\") pod \"dns-default-7qrj2\" (UID: \"13faa1a0-2c8b-4bb0-9531-10462aa1c3f4\") " pod="openshift-dns/dns-default-7qrj2"
Oct 31 22:49:50.044157 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.044135    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5d845527-5948-4edd-baab-2fe6d2bcba4a-bound-sa-token\") pod \"image-registry-79bdd8d85d-qbfpp\" (UID: \"5d845527-5948-4edd-baab-2fe6d2bcba4a\") " pod="openshift-image-registry/image-registry-79bdd8d85d-qbfpp"
Oct 31 22:49:50.068108 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.068079    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gcst4-bgj2j"
Oct 31 22:49:50.084117 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.084097    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5698547f59-qrxbm"
Oct 31 22:49:50.090968 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.090945    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-784qh"
Oct 31 22:49:50.101737 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.101717    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-n8vr2"
Oct 31 22:49:50.111440 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.111233    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t6v8n"
Oct 31 22:49:50.136155 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.135959    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mg6wn"
Oct 31 22:49:50.188875 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.187516    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7qrj2"
Oct 31 22:49:50.208082 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.207389    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79bdd8d85d-qbfpp"
Oct 31 22:49:50.352438 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.352407    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-gcst4-bgj2j"]
Oct 31 22:49:50.356495 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:50.356467    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3fb9361_cdf3_40d3_ae4b_d01fb6a569d0.slice/crio-a93270dfd699138c6bc985c31777bcf8a5ffba0195c49b34e8a2a0e767b4e529 WatchSource:0}: Error finding container a93270dfd699138c6bc985c31777bcf8a5ffba0195c49b34e8a2a0e767b4e529: Status 404 returned error can't find the container with id a93270dfd699138c6bc985c31777bcf8a5ffba0195c49b34e8a2a0e767b4e529
Oct 31 22:49:50.372901 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.372873    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5698547f59-qrxbm"]
Oct 31 22:49:50.377565 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:50.377541    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod593497dc_c158_470b_9202_b81c33373e33.slice/crio-e9c62a34a96681cfe25f3c0d94697a968fd940403e2c99d990ecafaa6a77bc26 WatchSource:0}: Error finding container e9c62a34a96681cfe25f3c0d94697a968fd940403e2c99d990ecafaa6a77bc26: Status 404 returned error can't find the container with id e9c62a34a96681cfe25f3c0d94697a968fd940403e2c99d990ecafaa6a77bc26
Oct 31 22:49:50.401232 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.401156    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-784qh"]
Oct 31 22:49:50.404968 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:50.404937    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3625be12_bd2d_4134_b3a9_4f67638d0c4e.slice/crio-cf451f040a2934c76297664a25234ecefab80c1759bed7e3c4d7cbd001a73899 WatchSource:0}: Error finding container cf451f040a2934c76297664a25234ecefab80c1759bed7e3c4d7cbd001a73899: Status 404 returned error can't find the container with id cf451f040a2934c76297664a25234ecefab80c1759bed7e3c4d7cbd001a73899
Oct 31 22:49:50.422960 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.422935    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-n8vr2"]
Oct 31 22:49:50.426601 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:50.426576    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca495489_b79a_45ad_b566_68c7090b04c9.slice/crio-6db451b5f58525f1ba3e4fe9a7364af3335cdde5d3f0431c688169c86c8112ee WatchSource:0}: Error finding container 6db451b5f58525f1ba3e4fe9a7364af3335cdde5d3f0431c688169c86c8112ee: Status 404 returned error can't find the container with id 6db451b5f58525f1ba3e4fe9a7364af3335cdde5d3f0431c688169c86c8112ee
Oct 31 22:49:50.438613 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.438588    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t6v8n"]
Oct 31 22:49:50.442700 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:50.442677    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2425ea7b_ae01_49db_950b_8f5268a4c0d5.slice/crio-04c441326ee45f7ed7c461b0ab926aca25f326389940bdba812598f162775a1f WatchSource:0}: Error finding container 04c441326ee45f7ed7c461b0ab926aca25f326389940bdba812598f162775a1f: Status 404 returned error can't find the container with id 04c441326ee45f7ed7c461b0ab926aca25f326389940bdba812598f162775a1f
Oct 31 22:49:50.451203 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.451178    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t6v8n" event={"ID":"2425ea7b-ae01-49db-950b-8f5268a4c0d5","Type":"ContainerStarted","Data":"04c441326ee45f7ed7c461b0ab926aca25f326389940bdba812598f162775a1f"}
Oct 31 22:49:50.452522 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.452496    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-n8vr2" event={"ID":"ca495489-b79a-45ad-b566-68c7090b04c9","Type":"ContainerStarted","Data":"6db451b5f58525f1ba3e4fe9a7364af3335cdde5d3f0431c688169c86c8112ee"}
Oct 31 22:49:50.453660 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.453627    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-784qh" event={"ID":"3625be12-bd2d-4134-b3a9-4f67638d0c4e","Type":"ContainerStarted","Data":"cf451f040a2934c76297664a25234ecefab80c1759bed7e3c4d7cbd001a73899"}
Oct 31 22:49:50.454757 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.454737    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5698547f59-qrxbm" event={"ID":"593497dc-c158-470b-9202-b81c33373e33","Type":"ContainerStarted","Data":"e9c62a34a96681cfe25f3c0d94697a968fd940403e2c99d990ecafaa6a77bc26"}
Oct 31 22:49:50.455724 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.455705    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gcst4-bgj2j" event={"ID":"a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0","Type":"ContainerStarted","Data":"a93270dfd699138c6bc985c31777bcf8a5ffba0195c49b34e8a2a0e767b4e529"}
Oct 31 22:49:50.473400 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.473205    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mg6wn"]
Oct 31 22:49:50.478322 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:50.478298    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a85016a_a525_4512_a83f_b355b784b58e.slice/crio-be9671e89c6ab8bffe88f2cc73338190fc98213c5c4adf80f9ebb60d8381736a WatchSource:0}: Error finding container be9671e89c6ab8bffe88f2cc73338190fc98213c5c4adf80f9ebb60d8381736a: Status 404 returned error can't find the container with id be9671e89c6ab8bffe88f2cc73338190fc98213c5c4adf80f9ebb60d8381736a
Oct 31 22:49:50.493385 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.492480    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7qrj2"]
Oct 31 22:49:50.502290 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:50.502267    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13faa1a0_2c8b_4bb0_9531_10462aa1c3f4.slice/crio-7f92c1baba73a465cbf558bb5a2467b5f059945c22eaf7de61fdadcfe08e5a85 WatchSource:0}: Error finding container 7f92c1baba73a465cbf558bb5a2467b5f059945c22eaf7de61fdadcfe08e5a85: Status 404 returned error can't find the container with id 7f92c1baba73a465cbf558bb5a2467b5f059945c22eaf7de61fdadcfe08e5a85
Oct 31 22:49:50.518931 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:50.518883    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79bdd8d85d-qbfpp"]
Oct 31 22:49:50.522775 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:50.522751    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d845527_5948_4edd_baab_2fe6d2bcba4a.slice/crio-674b5697765e713ff72d8b5b9f5757452fb6c528ef723a8e11de1bd3de738783 WatchSource:0}: Error finding container 674b5697765e713ff72d8b5b9f5757452fb6c528ef723a8e11de1bd3de738783: Status 404 returned error can't find the container with id 674b5697765e713ff72d8b5b9f5757452fb6c528ef723a8e11de1bd3de738783
Oct 31 22:49:51.463702 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:51.463667    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79bdd8d85d-qbfpp" event={"ID":"5d845527-5948-4edd-baab-2fe6d2bcba4a","Type":"ContainerStarted","Data":"599b9cf1a740f8cac54c2a90478c23a182c3d31a344fbcf7a8944829e91c7c91"}
Oct 31 22:49:51.464212 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:51.463726    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79bdd8d85d-qbfpp" event={"ID":"5d845527-5948-4edd-baab-2fe6d2bcba4a","Type":"ContainerStarted","Data":"674b5697765e713ff72d8b5b9f5757452fb6c528ef723a8e11de1bd3de738783"}
Oct 31 22:49:51.465737 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:51.465711    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7qrj2" event={"ID":"13faa1a0-2c8b-4bb0-9531-10462aa1c3f4","Type":"ContainerStarted","Data":"7f92c1baba73a465cbf558bb5a2467b5f059945c22eaf7de61fdadcfe08e5a85"}
Oct 31 22:49:51.475079 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:51.475051    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mg6wn" event={"ID":"2a85016a-a525-4512-a83f-b355b784b58e","Type":"ContainerStarted","Data":"ed06b8d724152404eaebf5bd7201e959f6031f8d5e750855fdbc0728d3266c5f"}
Oct 31 22:49:51.475194 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:51.475105    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mg6wn" event={"ID":"2a85016a-a525-4512-a83f-b355b784b58e","Type":"ContainerStarted","Data":"89002fe2af7773b04f06427cb78b57d7000c1f46919a99c9a9d9773029a82333"}
Oct 31 22:49:51.475194 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:51.475134    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mg6wn" event={"ID":"2a85016a-a525-4512-a83f-b355b784b58e","Type":"ContainerStarted","Data":"be9671e89c6ab8bffe88f2cc73338190fc98213c5c4adf80f9ebb60d8381736a"}
Oct 31 22:49:51.492845 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:51.491556    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-79bdd8d85d-qbfpp" podStartSLOduration=2.491530488 podStartE2EDuration="2.491530488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:51.490645985 +0000 UTC m=+10.237835490" watchObservedRunningTime="2026-10-31 22:49:51.491530488 +0000 UTC m=+10.238719990"
Oct 31 22:49:52.490492 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:52.488237    4301 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 22:49:52.490492 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:52.488854    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-784qh" event={"ID":"3625be12-bd2d-4134-b3a9-4f67638d0c4e","Type":"ContainerStarted","Data":"f3268085ef1c7acceca4c6bd60ec8b4376289a0c0dd467415954d756ed4a9012"}
Oct 31 22:49:52.490492 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:52.489153    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-784qh"
Oct 31 22:49:52.496679 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:52.496618    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-784qh"
Oct 31 22:49:52.508344 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:52.508020    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-784qh" podStartSLOduration=1.603695351 podStartE2EDuration="3.508004764s" totalImagesPullingTime="1.904309413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:52.507630282 +0000 UTC m=+11.254819787" watchObservedRunningTime="2026-10-31 22:49:52.508004764 +0000 UTC m=+11.255194269"
Oct 31 22:49:53.103388 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:53.103258    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79bdd8d85d-qbfpp"
Oct 31 22:49:53.433053 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:53.433012    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-spzc7"]
Oct 31 22:49:53.437779 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:53.437755    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-spzc7"
Oct 31 22:49:53.442699 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:53.442595    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 22:49:53.442699 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:53.442638    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 31 22:49:53.442699 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:53.442595    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 31 22:49:53.442940 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:53.442737    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-dzlf6\""
Oct 31 22:49:53.442940 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:53.442830    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 22:49:53.444453 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:53.444403    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 22:49:53.447012 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:53.446794    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-spzc7"]
Oct 31 22:49:53.561546 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:53.561498    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrpnp\" (UniqueName: \"kubernetes.io/projected/071935de-a5ac-4642-9f27-209332334466-kube-api-access-jrpnp\") pod \"prometheus-operator-9cd89b74c-spzc7\" (UID: \"071935de-a5ac-4642-9f27-209332334466\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-spzc7"
Oct 31 22:49:53.562267 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:53.561622    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/071935de-a5ac-4642-9f27-209332334466-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-spzc7\" (UID: \"071935de-a5ac-4642-9f27-209332334466\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-spzc7"
Oct 31 22:49:53.562267 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:53.561735    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/071935de-a5ac-4642-9f27-209332334466-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-spzc7\" (UID: \"071935de-a5ac-4642-9f27-209332334466\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-spzc7"
Oct 31 22:49:53.562267 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:53.561769    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/071935de-a5ac-4642-9f27-209332334466-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-spzc7\" (UID: \"071935de-a5ac-4642-9f27-209332334466\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-spzc7"
Oct 31 22:49:53.662845 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:53.662786    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/071935de-a5ac-4642-9f27-209332334466-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-spzc7\" (UID: \"071935de-a5ac-4642-9f27-209332334466\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-spzc7"
Oct 31 22:49:53.662845 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:53.662837    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/071935de-a5ac-4642-9f27-209332334466-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-spzc7\" (UID: \"071935de-a5ac-4642-9f27-209332334466\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-spzc7"
Oct 31 22:49:53.663099 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:53.662873    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrpnp\" (UniqueName: \"kubernetes.io/projected/071935de-a5ac-4642-9f27-209332334466-kube-api-access-jrpnp\") pod \"prometheus-operator-9cd89b74c-spzc7\" (UID: \"071935de-a5ac-4642-9f27-209332334466\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-spzc7"
Oct 31 22:49:53.663099 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:53.662931    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/071935de-a5ac-4642-9f27-209332334466-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-spzc7\" (UID: \"071935de-a5ac-4642-9f27-209332334466\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-spzc7"
Oct 31 22:49:53.664238 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:53.664183    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/071935de-a5ac-4642-9f27-209332334466-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-spzc7\" (UID: \"071935de-a5ac-4642-9f27-209332334466\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-spzc7"
Oct 31 22:49:53.671227 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:53.671174    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/071935de-a5ac-4642-9f27-209332334466-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-spzc7\" (UID: \"071935de-a5ac-4642-9f27-209332334466\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-spzc7"
Oct 31 22:49:53.674525 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:53.674471    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/071935de-a5ac-4642-9f27-209332334466-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-spzc7\" (UID: \"071935de-a5ac-4642-9f27-209332334466\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-spzc7"
Oct 31 22:49:53.683981 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:53.683918    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrpnp\" (UniqueName: \"kubernetes.io/projected/071935de-a5ac-4642-9f27-209332334466-kube-api-access-jrpnp\") pod \"prometheus-operator-9cd89b74c-spzc7\" (UID: \"071935de-a5ac-4642-9f27-209332334466\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-spzc7"
Oct 31 22:49:53.760453 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:53.760412    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-spzc7"
Oct 31 22:49:54.065489 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:54.065462    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-spzc7"]
Oct 31 22:49:54.499725 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:54.499639    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-spzc7" event={"ID":"071935de-a5ac-4642-9f27-209332334466","Type":"ContainerStarted","Data":"4dc0ab88c2f130e343f59686baaaa1ff6ec0460b3119c12a88cc36b937387331"}
Oct 31 22:49:54.501910 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:54.501879    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gcst4-bgj2j" event={"ID":"a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0","Type":"ContainerStarted","Data":"05cdaf564b576435661180c0199b156cb615a7a72001067502c6502284138400"}
Oct 31 22:49:54.507379 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:54.506906    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7qrj2" event={"ID":"13faa1a0-2c8b-4bb0-9531-10462aa1c3f4","Type":"ContainerStarted","Data":"1f3df8c688b563ecb613037127843a0a3d6f70b7d9b6562e5686252ce463265d"}
Oct 31 22:49:54.507379 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:54.506958    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7qrj2" event={"ID":"13faa1a0-2c8b-4bb0-9531-10462aa1c3f4","Type":"ContainerStarted","Data":"88f282d3bce82cdb7504158e5a54c60228174c392b756ded4c49f98443ff7711"}
Oct 31 22:49:54.507379 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:54.506991    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7qrj2"
Oct 31 22:49:54.512979 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:54.512898    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mg6wn" event={"ID":"2a85016a-a525-4512-a83f-b355b784b58e","Type":"ContainerStarted","Data":"c26c6415969640d6cb5e94ed1ae29512d9e8eb2021fdb838a98f9059bf72a4b5"}
Oct 31 22:49:54.542865 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:54.542805    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-gcst4-bgj2j" podStartSLOduration=1.591416858 podStartE2EDuration="5.542787726s" totalImagesPullingTime="3.951370868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:54.520638305 +0000 UTC m=+13.267827815" watchObservedRunningTime="2026-10-31 22:49:54.542787726 +0000 UTC m=+13.289977233"
Oct 31 22:49:54.543580 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:54.543522    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-7qrj2" podStartSLOduration=2.129049219 podStartE2EDuration="5.543511718s" totalImagesPullingTime="3.414462499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:54.542334902 +0000 UTC m=+13.289524405" watchObservedRunningTime="2026-10-31 22:49:54.543511718 +0000 UTC m=+13.290701260"
Oct 31 22:49:54.564193 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:54.564147    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mg6wn" podStartSLOduration=2.237788069 podStartE2EDuration="5.56413309s" totalImagesPullingTime="3.326345021s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:49:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:54.563721137 +0000 UTC m=+13.310910642" watchObservedRunningTime="2026-10-31 22:49:54.56413309 +0000 UTC m=+13.311322594"
Oct 31 22:49:55.189611 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:55.189579    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7qrj2_13faa1a0-2c8b-4bb0-9531-10462aa1c3f4/dns/0.log"
Oct 31 22:49:55.356792 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:55.356759    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7qrj2_13faa1a0-2c8b-4bb0-9531-10462aa1c3f4/kube-rbac-proxy/0.log"
Oct 31 22:49:55.519199 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:55.519117    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t6v8n" event={"ID":"2425ea7b-ae01-49db-950b-8f5268a4c0d5","Type":"ContainerStarted","Data":"55121a140d945aafa99fc40f261a639d6ba66d4eeff94bd4db58c05865bbaebc"}
Oct 31 22:49:55.960596 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:55.960577    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cn8jx_e46ea70b-37a4-4934-920c-ecdab212ecc6/dns-node-resolver/0.log"
Oct 31 22:49:56.524675 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:56.524637    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-spzc7" event={"ID":"071935de-a5ac-4642-9f27-209332334466","Type":"ContainerStarted","Data":"0a76eb55e2e67796fdd886132416f2a9402f150132d4ba0604543acde03b7fca"}
Oct 31 22:49:56.524857 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:56.524704    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-spzc7" event={"ID":"071935de-a5ac-4642-9f27-209332334466","Type":"ContainerStarted","Data":"78640fbf4207322f7922fdd6b86ba4142d985ec5a3221f2df24af94c01afb9c6"}
Oct 31 22:49:56.526893 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:56.526864    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5698547f59-qrxbm" event={"ID":"593497dc-c158-470b-9202-b81c33373e33","Type":"ContainerStarted","Data":"caa479c48156e11970e800f2e3f4a639e48c8c19b775205cabfe09a344994009"}
Oct 31 22:49:56.559271 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:56.559208    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-t6v8n" podStartSLOduration=2.8056012040000002 podStartE2EDuration="7.559191772s" totalImagesPullingTime="4.753590568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:55.538999352 +0000 UTC m=+14.286188857" watchObservedRunningTime="2026-10-31 22:49:56.559191772 +0000 UTC m=+15.306381276"
Oct 31 22:49:56.559579 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:56.559551    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-spzc7" podStartSLOduration=1.817843244 podStartE2EDuration="3.559542431s" totalImagesPullingTime="1.741699187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:49:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:56.558427636 +0000 UTC m=+15.305617138" watchObservedRunningTime="2026-10-31 22:49:56.559542431 +0000 UTC m=+15.306731934"
Oct 31 22:49:56.577874 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:56.577815    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5698547f59-qrxbm" podStartSLOduration=2.199283691 podStartE2EDuration="7.577803118s" totalImagesPullingTime="5.378519427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:56.577238088 +0000 UTC m=+15.324427593" watchObservedRunningTime="2026-10-31 22:49:56.577803118 +0000 UTC m=+15.324992623"
Oct 31 22:49:56.957176 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:56.957146    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-79bdd8d85d-qbfpp_5d845527-5948-4edd-baab-2fe6d2bcba4a/registry/0.log"
Oct 31 22:49:57.556281 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:57.556252    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xw8kl_8b1db31b-aec0-495b-8b0c-4abbde3acb28/node-ca/0.log"
Oct 31 22:49:58.157215 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:58.157186    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-t6v8n_2425ea7b-ae01-49db-950b-8f5268a4c0d5/serve-healthcheck-canary/0.log"
Oct 31 22:49:58.837041 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:58.837010    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jsnjt"]
Oct 31 22:49:58.840785 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:58.840759    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jsnjt"
Oct 31 22:49:58.881310 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:58.881268    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 22:49:58.881310 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:58.881304    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 22:49:58.881590 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:58.881477    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-p9kg6\""
Oct 31 22:49:58.914240 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:58.914203    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4f04cc51-6c46-481c-a5e3-367e11024767-node-exporter-tls\") pod \"node-exporter-jsnjt\" (UID: \"4f04cc51-6c46-481c-a5e3-367e11024767\") " pod="openshift-monitoring/node-exporter-jsnjt"
Oct 31 22:49:58.914420 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:58.914249    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f04cc51-6c46-481c-a5e3-367e11024767-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jsnjt\" (UID: \"4f04cc51-6c46-481c-a5e3-367e11024767\") " pod="openshift-monitoring/node-exporter-jsnjt"
Oct 31 22:49:58.914500 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:58.914443    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4f04cc51-6c46-481c-a5e3-367e11024767-node-exporter-accelerators-collector-config\") pod \"node-exporter-jsnjt\" (UID: \"4f04cc51-6c46-481c-a5e3-367e11024767\") " pod="openshift-monitoring/node-exporter-jsnjt"
Oct 31 22:49:58.914500 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:58.914472    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f04cc51-6c46-481c-a5e3-367e11024767-metrics-client-ca\") pod \"node-exporter-jsnjt\" (UID: \"4f04cc51-6c46-481c-a5e3-367e11024767\") " pod="openshift-monitoring/node-exporter-jsnjt"
Oct 31 22:49:58.914585 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:58.914498    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmqrk\" (UniqueName: \"kubernetes.io/projected/4f04cc51-6c46-481c-a5e3-367e11024767-kube-api-access-pmqrk\") pod \"node-exporter-jsnjt\" (UID: \"4f04cc51-6c46-481c-a5e3-367e11024767\") " pod="openshift-monitoring/node-exporter-jsnjt"
Oct 31 22:49:58.914630 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:58.914617    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4f04cc51-6c46-481c-a5e3-367e11024767-root\") pod \"node-exporter-jsnjt\" (UID: \"4f04cc51-6c46-481c-a5e3-367e11024767\") " pod="openshift-monitoring/node-exporter-jsnjt"
Oct 31 22:49:58.914672 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:58.914643    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4f04cc51-6c46-481c-a5e3-367e11024767-node-exporter-wtmp\") pod \"node-exporter-jsnjt\" (UID: \"4f04cc51-6c46-481c-a5e3-367e11024767\") " pod="openshift-monitoring/node-exporter-jsnjt"
Oct 31 22:49:58.914722 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:58.914702    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f04cc51-6c46-481c-a5e3-367e11024767-sys\") pod \"node-exporter-jsnjt\" (UID: \"4f04cc51-6c46-481c-a5e3-367e11024767\") " pod="openshift-monitoring/node-exporter-jsnjt"
Oct 31 22:49:58.914771 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:58.914738    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4f04cc51-6c46-481c-a5e3-367e11024767-node-exporter-textfile\") pod \"node-exporter-jsnjt\" (UID: \"4f04cc51-6c46-481c-a5e3-367e11024767\") " pod="openshift-monitoring/node-exporter-jsnjt"
Oct 31 22:49:58.924286 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:58.924250    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 22:49:59.015687 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:59.015656    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4f04cc51-6c46-481c-a5e3-367e11024767-node-exporter-accelerators-collector-config\") pod \"node-exporter-jsnjt\" (UID: \"4f04cc51-6c46-481c-a5e3-367e11024767\") " pod="openshift-monitoring/node-exporter-jsnjt"
Oct 31 22:49:59.015687 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:59.015694    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f04cc51-6c46-481c-a5e3-367e11024767-metrics-client-ca\") pod \"node-exporter-jsnjt\" (UID: \"4f04cc51-6c46-481c-a5e3-367e11024767\") " pod="openshift-monitoring/node-exporter-jsnjt"
Oct 31 22:49:59.015939 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:59.015732    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmqrk\" (UniqueName: \"kubernetes.io/projected/4f04cc51-6c46-481c-a5e3-367e11024767-kube-api-access-pmqrk\") pod \"node-exporter-jsnjt\" (UID: \"4f04cc51-6c46-481c-a5e3-367e11024767\") " pod="openshift-monitoring/node-exporter-jsnjt"
Oct 31 22:49:59.015939 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:59.015776    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4f04cc51-6c46-481c-a5e3-367e11024767-root\") pod \"node-exporter-jsnjt\" (UID: \"4f04cc51-6c46-481c-a5e3-367e11024767\") " pod="openshift-monitoring/node-exporter-jsnjt"
Oct 31 22:49:59.015939 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:59.015798    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4f04cc51-6c46-481c-a5e3-367e11024767-node-exporter-wtmp\") pod \"node-exporter-jsnjt\" (UID: \"4f04cc51-6c46-481c-a5e3-367e11024767\") " pod="openshift-monitoring/node-exporter-jsnjt"
Oct 31 22:49:59.015939 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:59.015826    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f04cc51-6c46-481c-a5e3-367e11024767-sys\") pod \"node-exporter-jsnjt\" (UID: \"4f04cc51-6c46-481c-a5e3-367e11024767\") " pod="openshift-monitoring/node-exporter-jsnjt"
Oct 31 22:49:59.015939 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:59.015848    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4f04cc51-6c46-481c-a5e3-367e11024767-node-exporter-textfile\") pod \"node-exporter-jsnjt\" (UID: \"4f04cc51-6c46-481c-a5e3-367e11024767\") " pod="openshift-monitoring/node-exporter-jsnjt"
Oct 31 22:49:59.015939 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:59.015872    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4f04cc51-6c46-481c-a5e3-367e11024767-root\") pod \"node-exporter-jsnjt\" (UID: \"4f04cc51-6c46-481c-a5e3-367e11024767\") " pod="openshift-monitoring/node-exporter-jsnjt"
Oct 31 22:49:59.016218 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:59.016000    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4f04cc51-6c46-481c-a5e3-367e11024767-node-exporter-wtmp\") pod \"node-exporter-jsnjt\" (UID: \"4f04cc51-6c46-481c-a5e3-367e11024767\") " pod="openshift-monitoring/node-exporter-jsnjt"
Oct 31 22:49:59.016218 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:59.015881    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4f04cc51-6c46-481c-a5e3-367e11024767-node-exporter-tls\") pod \"node-exporter-jsnjt\" (UID: \"4f04cc51-6c46-481c-a5e3-367e11024767\") " pod="openshift-monitoring/node-exporter-jsnjt"
Oct 31 22:49:59.016322 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:59.016224    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f04cc51-6c46-481c-a5e3-367e11024767-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jsnjt\" (UID: \"4f04cc51-6c46-481c-a5e3-367e11024767\") " pod="openshift-monitoring/node-exporter-jsnjt"
Oct 31 22:49:59.016680 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:59.016435    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f04cc51-6c46-481c-a5e3-367e11024767-sys\") pod \"node-exporter-jsnjt\" (UID: \"4f04cc51-6c46-481c-a5e3-367e11024767\") " pod="openshift-monitoring/node-exporter-jsnjt"
Oct 31 22:49:59.016680 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:59.016472    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f04cc51-6c46-481c-a5e3-367e11024767-metrics-client-ca\") pod \"node-exporter-jsnjt\" (UID: \"4f04cc51-6c46-481c-a5e3-367e11024767\") " pod="openshift-monitoring/node-exporter-jsnjt"
Oct 31 22:49:59.016680 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:59.016493    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4f04cc51-6c46-481c-a5e3-367e11024767-node-exporter-accelerators-collector-config\") pod \"node-exporter-jsnjt\" (UID: \"4f04cc51-6c46-481c-a5e3-367e11024767\") " pod="openshift-monitoring/node-exporter-jsnjt"
Oct 31 22:49:59.016680 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:59.016642    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4f04cc51-6c46-481c-a5e3-367e11024767-node-exporter-textfile\") pod \"node-exporter-jsnjt\" (UID: \"4f04cc51-6c46-481c-a5e3-367e11024767\") " pod="openshift-monitoring/node-exporter-jsnjt"
Oct 31 22:49:59.018750 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:59.018709    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f04cc51-6c46-481c-a5e3-367e11024767-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jsnjt\" (UID: \"4f04cc51-6c46-481c-a5e3-367e11024767\") " pod="openshift-monitoring/node-exporter-jsnjt"
Oct 31 22:49:59.023602 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:59.023582    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4f04cc51-6c46-481c-a5e3-367e11024767-node-exporter-tls\") pod \"node-exporter-jsnjt\" (UID: \"4f04cc51-6c46-481c-a5e3-367e11024767\") " pod="openshift-monitoring/node-exporter-jsnjt"
Oct 31 22:49:59.028256 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:59.028230    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmqrk\" (UniqueName: \"kubernetes.io/projected/4f04cc51-6c46-481c-a5e3-367e11024767-kube-api-access-pmqrk\") pod \"node-exporter-jsnjt\" (UID: \"4f04cc51-6c46-481c-a5e3-367e11024767\") " pod="openshift-monitoring/node-exporter-jsnjt"
Oct 31 22:49:59.158135 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:59.158097    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jsnjt"
Oct 31 22:49:59.170586 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:49:59.170548    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f04cc51_6c46_481c_a5e3_367e11024767.slice/crio-66fd14412f13306d0c86ad16cee7f0f68b28dc697173ee14564a0927186903cb WatchSource:0}: Error finding container 66fd14412f13306d0c86ad16cee7f0f68b28dc697173ee14564a0927186903cb: Status 404 returned error can't find the container with id 66fd14412f13306d0c86ad16cee7f0f68b28dc697173ee14564a0927186903cb
Oct 31 22:49:59.543521 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:49:59.543439    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jsnjt" event={"ID":"4f04cc51-6c46-481c-a5e3-367e11024767","Type":"ContainerStarted","Data":"66fd14412f13306d0c86ad16cee7f0f68b28dc697173ee14564a0927186903cb"}
Oct 31 22:50:00.549800 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:00.549765    4301 generic.go:356] "Generic (PLEG): container finished" podID="4f04cc51-6c46-481c-a5e3-367e11024767" containerID="5422b41477d0368c7b2913326842f7c5952b187dba4b71fea33125f120cb231c" exitCode=0
Oct 31 22:50:00.550219 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:00.549819    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jsnjt" event={"ID":"4f04cc51-6c46-481c-a5e3-367e11024767","Type":"ContainerDied","Data":"5422b41477d0368c7b2913326842f7c5952b187dba4b71fea33125f120cb231c"}
Oct 31 22:50:00.698865 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:00.698839    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5698547f59-qrxbm"
Oct 31 22:50:00.705165 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:00.705140    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5698547f59-qrxbm"
Oct 31 22:50:01.315367 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:01.315321    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5698547f59-qrxbm"
Oct 31 22:50:01.319501 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:01.319468    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5698547f59-qrxbm"
Oct 31 22:50:01.557464 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:01.557427    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jsnjt" event={"ID":"4f04cc51-6c46-481c-a5e3-367e11024767","Type":"ContainerStarted","Data":"4f8004a3cdc259b339404be8e4d545adbf4d0db2096069642c93aa7305eb3cfc"}
Oct 31 22:50:01.557905 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:01.557492    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jsnjt" event={"ID":"4f04cc51-6c46-481c-a5e3-367e11024767","Type":"ContainerStarted","Data":"a0c3ead7377cd02adbdac2967ab6901a6c6f8c2898d8a2f0429c26411f33b3eb"}
Oct 31 22:50:01.629792 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:01.629728    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jsnjt" podStartSLOduration=2.415825485 podStartE2EDuration="3.629709653s" totalImagesPullingTime="1.136552752s" totalInitContainerRuntime="77.331416ms" isStatefulPod=true podCreationTimestamp="2026-10-31 22:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:50:01.626241663 +0000 UTC m=+20.373431170" watchObservedRunningTime="2026-10-31 22:50:01.629709653 +0000 UTC m=+20.376899158"
Oct 31 22:50:02.547877 ip-10-0-135-163 kubenswrapper[4301]: E1031 22:50:02.547841    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 22:50:04.523977 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:04.523945    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7qrj2"
Oct 31 22:50:09.591087 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:09.591036    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-n8vr2" event={"ID":"ca495489-b79a-45ad-b566-68c7090b04c9","Type":"ContainerStarted","Data":"e765565a04a381f06327b1631ee4d1247a90503f4bcb024fcfd1899ce4d11b0b"}
Oct 31 22:50:09.591697 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:09.591519    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-n8vr2"
Oct 31 22:50:09.608228 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:09.608200    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-n8vr2"
Oct 31 22:50:09.611601 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:09.611558    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-n8vr2" podStartSLOduration=2.050480934 podStartE2EDuration="20.611546442s" totalImagesPullingTime="18.561065508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:50:09.610591854 +0000 UTC m=+28.357781361" watchObservedRunningTime="2026-10-31 22:50:09.611546442 +0000 UTC m=+28.358735971"
Oct 31 22:50:13.436491 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:13.436434    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ddxkp"
Oct 31 22:50:13.499014 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:13.498983    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79bdd8d85d-qbfpp"
Oct 31 22:50:14.661351 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:14.661317    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5698547f59-qrxbm"]
Oct 31 22:50:32.544782 ip-10-0-135-163 kubenswrapper[4301]: E1031 22:50:32.544689    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 22:50:39.686809 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:39.686762    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5698547f59-qrxbm" podUID="593497dc-c158-470b-9202-b81c33373e33" containerName="console" containerID="cri-o://caa479c48156e11970e800f2e3f4a639e48c8c19b775205cabfe09a344994009" gracePeriod=15
Oct 31 22:50:39.966148 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:39.966085    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5698547f59-qrxbm"
Oct 31 22:50:40.031915 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:40.031882    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/593497dc-c158-470b-9202-b81c33373e33-tmp\" (UniqueName: \"kubernetes.io/empty-dir/593497dc-c158-470b-9202-b81c33373e33-tmp\") pod \"593497dc-c158-470b-9202-b81c33373e33\" (UID: \"593497dc-c158-470b-9202-b81c33373e33\") "
Oct 31 22:50:40.031915 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:40.031921    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/593497dc-c158-470b-9202-b81c33373e33-console-config\" (UniqueName: \"kubernetes.io/configmap/593497dc-c158-470b-9202-b81c33373e33-console-config\") pod \"593497dc-c158-470b-9202-b81c33373e33\" (UID: \"593497dc-c158-470b-9202-b81c33373e33\") "
Oct 31 22:50:40.032152 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:40.031953    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/593497dc-c158-470b-9202-b81c33373e33-kube-api-access-6s6b8\" (UniqueName: \"kubernetes.io/projected/593497dc-c158-470b-9202-b81c33373e33-kube-api-access-6s6b8\") pod \"593497dc-c158-470b-9202-b81c33373e33\" (UID: \"593497dc-c158-470b-9202-b81c33373e33\") "
Oct 31 22:50:40.032152 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:40.031996    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/593497dc-c158-470b-9202-b81c33373e33-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/593497dc-c158-470b-9202-b81c33373e33-console-serving-cert\") pod \"593497dc-c158-470b-9202-b81c33373e33\" (UID: \"593497dc-c158-470b-9202-b81c33373e33\") "
Oct 31 22:50:40.032152 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:40.032043    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/593497dc-c158-470b-9202-b81c33373e33-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/593497dc-c158-470b-9202-b81c33373e33-console-oauth-config\") pod \"593497dc-c158-470b-9202-b81c33373e33\" (UID: \"593497dc-c158-470b-9202-b81c33373e33\") "
Oct 31 22:50:40.032309 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:40.032203    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/593497dc-c158-470b-9202-b81c33373e33-tmp" pod "593497dc-c158-470b-9202-b81c33373e33" (UID: "593497dc-c158-470b-9202-b81c33373e33"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 22:50:40.032309 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:40.032211    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/593497dc-c158-470b-9202-b81c33373e33-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/593497dc-c158-470b-9202-b81c33373e33-oauth-serving-cert\") pod \"593497dc-c158-470b-9202-b81c33373e33\" (UID: \"593497dc-c158-470b-9202-b81c33373e33\") "
Oct 31 22:50:40.032309 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:40.032280    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/593497dc-c158-470b-9202-b81c33373e33-service-ca\" (UniqueName: \"kubernetes.io/configmap/593497dc-c158-470b-9202-b81c33373e33-service-ca\") pod \"593497dc-c158-470b-9202-b81c33373e33\" (UID: \"593497dc-c158-470b-9202-b81c33373e33\") "
Oct 31 22:50:40.032480 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:40.032340    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/593497dc-c158-470b-9202-b81c33373e33-console-config" pod "593497dc-c158-470b-9202-b81c33373e33" (UID: "593497dc-c158-470b-9202-b81c33373e33"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 22:50:40.032646 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:40.032628    4301 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/593497dc-c158-470b-9202-b81c33373e33-tmp\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 22:50:40.032700 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:40.032652    4301 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/593497dc-c158-470b-9202-b81c33373e33-console-config\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 22:50:40.032700 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:40.032652    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/593497dc-c158-470b-9202-b81c33373e33-oauth-serving-cert" pod "593497dc-c158-470b-9202-b81c33373e33" (UID: "593497dc-c158-470b-9202-b81c33373e33"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 22:50:40.032762 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:40.032734    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/593497dc-c158-470b-9202-b81c33373e33-service-ca" pod "593497dc-c158-470b-9202-b81c33373e33" (UID: "593497dc-c158-470b-9202-b81c33373e33"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 22:50:40.034491 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:40.034469    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/593497dc-c158-470b-9202-b81c33373e33-console-oauth-config" pod "593497dc-c158-470b-9202-b81c33373e33" (UID: "593497dc-c158-470b-9202-b81c33373e33"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 22:50:40.034606 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:40.034488    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/593497dc-c158-470b-9202-b81c33373e33-console-serving-cert" pod "593497dc-c158-470b-9202-b81c33373e33" (UID: "593497dc-c158-470b-9202-b81c33373e33"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 22:50:40.034606 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:40.034497    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/593497dc-c158-470b-9202-b81c33373e33-kube-api-access-6s6b8" pod "593497dc-c158-470b-9202-b81c33373e33" (UID: "593497dc-c158-470b-9202-b81c33373e33"). InnerVolumeSpecName "kube-api-access-6s6b8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 22:50:40.133611 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:40.133578    4301 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/593497dc-c158-470b-9202-b81c33373e33-oauth-serving-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 22:50:40.133611 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:40.133604    4301 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/593497dc-c158-470b-9202-b81c33373e33-service-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 22:50:40.133611 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:40.133613    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s6b8\" (UniqueName: \"kubernetes.io/projected/593497dc-c158-470b-9202-b81c33373e33-kube-api-access-6s6b8\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 22:50:40.133611 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:40.133623    4301 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/593497dc-c158-470b-9202-b81c33373e33-console-serving-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 22:50:40.133857 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:40.133632    4301 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/593497dc-c158-470b-9202-b81c33373e33-console-oauth-config\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 22:50:40.694881 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:40.694848    4301 generic.go:356] "Generic (PLEG): container finished" podID="593497dc-c158-470b-9202-b81c33373e33" containerID="caa479c48156e11970e800f2e3f4a639e48c8c19b775205cabfe09a344994009" exitCode=0
Oct 31 22:50:40.695288 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:40.694896    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5698547f59-qrxbm"
Oct 31 22:50:40.695288 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:40.694900    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5698547f59-qrxbm" event={"ID":"593497dc-c158-470b-9202-b81c33373e33","Type":"ContainerDied","Data":"caa479c48156e11970e800f2e3f4a639e48c8c19b775205cabfe09a344994009"}
Oct 31 22:50:40.695288 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:40.695023    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5698547f59-qrxbm" event={"ID":"593497dc-c158-470b-9202-b81c33373e33","Type":"ContainerDied","Data":"e9c62a34a96681cfe25f3c0d94697a968fd940403e2c99d990ecafaa6a77bc26"}
Oct 31 22:50:40.695288 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:40.695054    4301 scope.go:122] "RemoveContainer" containerID="caa479c48156e11970e800f2e3f4a639e48c8c19b775205cabfe09a344994009"
Oct 31 22:50:40.707271 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:40.707252    4301 scope.go:122] "RemoveContainer" containerID="caa479c48156e11970e800f2e3f4a639e48c8c19b775205cabfe09a344994009"
Oct 31 22:50:40.707595 ip-10-0-135-163 kubenswrapper[4301]: E1031 22:50:40.707571    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caa479c48156e11970e800f2e3f4a639e48c8c19b775205cabfe09a344994009\": container with ID starting with caa479c48156e11970e800f2e3f4a639e48c8c19b775205cabfe09a344994009 not found: ID does not exist" containerID="caa479c48156e11970e800f2e3f4a639e48c8c19b775205cabfe09a344994009"
Oct 31 22:50:40.707678 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:40.707607    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caa479c48156e11970e800f2e3f4a639e48c8c19b775205cabfe09a344994009"} err="failed to get container status \"caa479c48156e11970e800f2e3f4a639e48c8c19b775205cabfe09a344994009\": rpc error: code = NotFound desc = could not find container \"caa479c48156e11970e800f2e3f4a639e48c8c19b775205cabfe09a344994009\": container with ID starting with caa479c48156e11970e800f2e3f4a639e48c8c19b775205cabfe09a344994009 not found: ID does not exist"
Oct 31 22:50:40.717011 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:40.716989    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5698547f59-qrxbm"]
Oct 31 22:50:40.720456 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:40.720435    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5698547f59-qrxbm"]
Oct 31 22:50:41.400467 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:41.400436    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="593497dc-c158-470b-9202-b81c33373e33" path="/var/lib/kubelet/pods/593497dc-c158-470b-9202-b81c33373e33/volumes"
Oct 31 22:50:51.505661 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.505512    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-85cd978d69-hmq7q"]
Oct 31 22:50:51.506274 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.505931    4301 cpu_manager.go:414] "containerMap: removing container" podUID="593497dc-c158-470b-9202-b81c33373e33" containerName="console"
Oct 31 22:50:51.506274 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.505950    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="593497dc-c158-470b-9202-b81c33373e33" containerName="console"
Oct 31 22:50:51.506274 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.506028    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="593497dc-c158-470b-9202-b81c33373e33" containerName="console"
Oct 31 22:50:51.539147 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.539121    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85cd978d69-hmq7q"]
Oct 31 22:50:51.539300 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.539242    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85cd978d69-hmq7q"
Oct 31 22:50:51.543065 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.543043    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 31 22:50:51.543386 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.543367    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-cbxkt\""
Oct 31 22:50:51.543482 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.543368    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 31 22:50:51.543482 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.543394    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 31 22:50:51.543482 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.543432    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 31 22:50:51.543648 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.543538    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 31 22:50:51.545099 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.545082    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 31 22:50:51.616717 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.616683    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1cd78063-4659-48fc-8937-5db020ee1aa9-console-oauth-config\") pod \"console-85cd978d69-hmq7q\" (UID: \"1cd78063-4659-48fc-8937-5db020ee1aa9\") " pod="openshift-console/console-85cd978d69-hmq7q"
Oct 31 22:50:51.616886 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.616723    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cd78063-4659-48fc-8937-5db020ee1aa9-trusted-ca-bundle\") pod \"console-85cd978d69-hmq7q\" (UID: \"1cd78063-4659-48fc-8937-5db020ee1aa9\") " pod="openshift-console/console-85cd978d69-hmq7q"
Oct 31 22:50:51.616886 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.616752    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1cd78063-4659-48fc-8937-5db020ee1aa9-console-serving-cert\") pod \"console-85cd978d69-hmq7q\" (UID: \"1cd78063-4659-48fc-8937-5db020ee1aa9\") " pod="openshift-console/console-85cd978d69-hmq7q"
Oct 31 22:50:51.616972 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.616879    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1cd78063-4659-48fc-8937-5db020ee1aa9-oauth-serving-cert\") pod \"console-85cd978d69-hmq7q\" (UID: \"1cd78063-4659-48fc-8937-5db020ee1aa9\") " pod="openshift-console/console-85cd978d69-hmq7q"
Oct 31 22:50:51.616972 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.616953    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd78063-4659-48fc-8937-5db020ee1aa9-service-ca\") pod \"console-85cd978d69-hmq7q\" (UID: \"1cd78063-4659-48fc-8937-5db020ee1aa9\") " pod="openshift-console/console-85cd978d69-hmq7q"
Oct 31 22:50:51.617054 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.617003    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1cd78063-4659-48fc-8937-5db020ee1aa9-console-config\") pod \"console-85cd978d69-hmq7q\" (UID: \"1cd78063-4659-48fc-8937-5db020ee1aa9\") " pod="openshift-console/console-85cd978d69-hmq7q"
Oct 31 22:50:51.617054 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.617018    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z28wc\" (UniqueName: \"kubernetes.io/projected/1cd78063-4659-48fc-8937-5db020ee1aa9-kube-api-access-z28wc\") pod \"console-85cd978d69-hmq7q\" (UID: \"1cd78063-4659-48fc-8937-5db020ee1aa9\") " pod="openshift-console/console-85cd978d69-hmq7q"
Oct 31 22:50:51.617128 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.617090    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cd78063-4659-48fc-8937-5db020ee1aa9-tmp\") pod \"console-85cd978d69-hmq7q\" (UID: \"1cd78063-4659-48fc-8937-5db020ee1aa9\") " pod="openshift-console/console-85cd978d69-hmq7q"
Oct 31 22:50:51.717589 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.717549    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1cd78063-4659-48fc-8937-5db020ee1aa9-oauth-serving-cert\") pod \"console-85cd978d69-hmq7q\" (UID: \"1cd78063-4659-48fc-8937-5db020ee1aa9\") " pod="openshift-console/console-85cd978d69-hmq7q"
Oct 31 22:50:51.717589 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.717589    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd78063-4659-48fc-8937-5db020ee1aa9-service-ca\") pod \"console-85cd978d69-hmq7q\" (UID: \"1cd78063-4659-48fc-8937-5db020ee1aa9\") " pod="openshift-console/console-85cd978d69-hmq7q"
Oct 31 22:50:51.717829 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.717626    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1cd78063-4659-48fc-8937-5db020ee1aa9-console-config\") pod \"console-85cd978d69-hmq7q\" (UID: \"1cd78063-4659-48fc-8937-5db020ee1aa9\") " pod="openshift-console/console-85cd978d69-hmq7q"
Oct 31 22:50:51.717829 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.717650    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z28wc\" (UniqueName: \"kubernetes.io/projected/1cd78063-4659-48fc-8937-5db020ee1aa9-kube-api-access-z28wc\") pod \"console-85cd978d69-hmq7q\" (UID: \"1cd78063-4659-48fc-8937-5db020ee1aa9\") " pod="openshift-console/console-85cd978d69-hmq7q"
Oct 31 22:50:51.717829 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.717681    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cd78063-4659-48fc-8937-5db020ee1aa9-tmp\") pod \"console-85cd978d69-hmq7q\" (UID: \"1cd78063-4659-48fc-8937-5db020ee1aa9\") " pod="openshift-console/console-85cd978d69-hmq7q"
Oct 31 22:50:51.717829 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.717809    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1cd78063-4659-48fc-8937-5db020ee1aa9-console-oauth-config\") pod \"console-85cd978d69-hmq7q\" (UID: \"1cd78063-4659-48fc-8937-5db020ee1aa9\") " pod="openshift-console/console-85cd978d69-hmq7q"
Oct 31 22:50:51.718015 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.717871    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cd78063-4659-48fc-8937-5db020ee1aa9-trusted-ca-bundle\") pod \"console-85cd978d69-hmq7q\" (UID: \"1cd78063-4659-48fc-8937-5db020ee1aa9\") " pod="openshift-console/console-85cd978d69-hmq7q"
Oct 31 22:50:51.718015 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.717930    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1cd78063-4659-48fc-8937-5db020ee1aa9-console-serving-cert\") pod \"console-85cd978d69-hmq7q\" (UID: \"1cd78063-4659-48fc-8937-5db020ee1aa9\") " pod="openshift-console/console-85cd978d69-hmq7q"
Oct 31 22:50:51.718130 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.718046    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cd78063-4659-48fc-8937-5db020ee1aa9-tmp\") pod \"console-85cd978d69-hmq7q\" (UID: \"1cd78063-4659-48fc-8937-5db020ee1aa9\") " pod="openshift-console/console-85cd978d69-hmq7q"
Oct 31 22:50:51.718470 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.718442    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd78063-4659-48fc-8937-5db020ee1aa9-service-ca\") pod \"console-85cd978d69-hmq7q\" (UID: \"1cd78063-4659-48fc-8937-5db020ee1aa9\") " pod="openshift-console/console-85cd978d69-hmq7q"
Oct 31 22:50:51.718738 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.718716    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1cd78063-4659-48fc-8937-5db020ee1aa9-console-config\") pod \"console-85cd978d69-hmq7q\" (UID: \"1cd78063-4659-48fc-8937-5db020ee1aa9\") " pod="openshift-console/console-85cd978d69-hmq7q"
Oct 31 22:50:51.718792 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.718722    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1cd78063-4659-48fc-8937-5db020ee1aa9-oauth-serving-cert\") pod \"console-85cd978d69-hmq7q\" (UID: \"1cd78063-4659-48fc-8937-5db020ee1aa9\") " pod="openshift-console/console-85cd978d69-hmq7q"
Oct 31 22:50:51.718875 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.718856    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cd78063-4659-48fc-8937-5db020ee1aa9-trusted-ca-bundle\") pod \"console-85cd978d69-hmq7q\" (UID: \"1cd78063-4659-48fc-8937-5db020ee1aa9\") " pod="openshift-console/console-85cd978d69-hmq7q"
Oct 31 22:50:51.720370 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.720337    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1cd78063-4659-48fc-8937-5db020ee1aa9-console-serving-cert\") pod \"console-85cd978d69-hmq7q\" (UID: \"1cd78063-4659-48fc-8937-5db020ee1aa9\") " pod="openshift-console/console-85cd978d69-hmq7q"
Oct 31 22:50:51.720446 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.720410    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1cd78063-4659-48fc-8937-5db020ee1aa9-console-oauth-config\") pod \"console-85cd978d69-hmq7q\" (UID: \"1cd78063-4659-48fc-8937-5db020ee1aa9\") " pod="openshift-console/console-85cd978d69-hmq7q"
Oct 31 22:50:51.729637 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.729615    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z28wc\" (UniqueName: \"kubernetes.io/projected/1cd78063-4659-48fc-8937-5db020ee1aa9-kube-api-access-z28wc\") pod \"console-85cd978d69-hmq7q\" (UID: \"1cd78063-4659-48fc-8937-5db020ee1aa9\") " pod="openshift-console/console-85cd978d69-hmq7q"
Oct 31 22:50:51.861894 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:51.861811    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85cd978d69-hmq7q"
Oct 31 22:50:52.165711 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:52.165672    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85cd978d69-hmq7q"]
Oct 31 22:50:52.169657 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:50:52.169628    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd78063_4659_48fc_8937_5db020ee1aa9.slice/crio-813503e535873ee1ab4c931193d36047f47eb6c6c12e41831d54e33e577cf663 WatchSource:0}: Error finding container 813503e535873ee1ab4c931193d36047f47eb6c6c12e41831d54e33e577cf663: Status 404 returned error can't find the container with id 813503e535873ee1ab4c931193d36047f47eb6c6c12e41831d54e33e577cf663
Oct 31 22:50:52.733963 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:52.733932    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85cd978d69-hmq7q" event={"ID":"1cd78063-4659-48fc-8937-5db020ee1aa9","Type":"ContainerStarted","Data":"3aff5ca63fe2d83d5f87f33656ef716b2505310298774fd885ea3138b4157eba"}
Oct 31 22:50:52.734397 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:52.733983    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85cd978d69-hmq7q" event={"ID":"1cd78063-4659-48fc-8937-5db020ee1aa9","Type":"ContainerStarted","Data":"813503e535873ee1ab4c931193d36047f47eb6c6c12e41831d54e33e577cf663"}
Oct 31 22:50:52.757737 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:50:52.757679    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-85cd978d69-hmq7q" podStartSLOduration=1.757666644 podStartE2EDuration="1.757666644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:50:52.7566848 +0000 UTC m=+71.503874306" watchObservedRunningTime="2026-10-31 22:50:52.757666644 +0000 UTC m=+71.504856149"
Oct 31 22:51:01.862704 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:01.862658    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85cd978d69-hmq7q"
Oct 31 22:51:01.863195 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:01.862759    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85cd978d69-hmq7q"
Oct 31 22:51:01.867404 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:01.867380    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85cd978d69-hmq7q"
Oct 31 22:51:02.543768 ip-10-0-135-163 kubenswrapper[4301]: E1031 22:51:02.543727    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 22:51:02.767497 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:02.767471    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85cd978d69-hmq7q"
Oct 31 22:51:21.823183 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:21.823149    4301 generic.go:356] "Generic (PLEG): container finished" podID="a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0" containerID="05cdaf564b576435661180c0199b156cb615a7a72001067502c6502284138400" exitCode=0
Oct 31 22:51:21.823627 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:21.823231    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gcst4-bgj2j" event={"ID":"a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0","Type":"ContainerDied","Data":"05cdaf564b576435661180c0199b156cb615a7a72001067502c6502284138400"}
Oct 31 22:51:22.949029 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:22.949008    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gcst4-bgj2j"
Oct 31 22:51:23.047651 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:23.047619    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0-trusted-ca-bundle\") pod \"a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0\" (UID: \"a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0\") "
Oct 31 22:51:23.047844 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:23.047683    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0-service-ca-bundle\") pod \"a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0\" (UID: \"a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0\") "
Oct 31 22:51:23.047844 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:23.047706    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0-kube-api-access-7wq5r\" (UniqueName: \"kubernetes.io/projected/a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0-kube-api-access-7wq5r\") pod \"a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0\" (UID: \"a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0\") "
Oct 31 22:51:23.047844 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:23.047745    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0-archives-path\") pod \"a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0\" (UID: \"a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0\") "
Oct 31 22:51:23.048050 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:23.048019    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0-trusted-ca-bundle" pod "a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0" (UID: "a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 22:51:23.048212 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:23.048057    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0-service-ca-bundle" pod "a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0" (UID: "a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 22:51:23.048212 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:23.048127    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0-archives-path" pod "a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0" (UID: "a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 22:51:23.049965 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:23.049942    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0-kube-api-access-7wq5r" pod "a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0" (UID: "a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0"). InnerVolumeSpecName "kube-api-access-7wq5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 22:51:23.148515 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:23.148478    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wq5r\" (UniqueName: \"kubernetes.io/projected/a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0-kube-api-access-7wq5r\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 22:51:23.148515 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:23.148516    4301 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0-archives-path\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 22:51:23.148734 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:23.148529    4301 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0-trusted-ca-bundle\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 22:51:23.148734 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:23.148556    4301 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0-service-ca-bundle\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 22:51:23.830385 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:23.830286    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gcst4-bgj2j" event={"ID":"a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0","Type":"ContainerDied","Data":"a93270dfd699138c6bc985c31777bcf8a5ffba0195c49b34e8a2a0e767b4e529"}
Oct 31 22:51:23.830385 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:23.830317    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gcst4-bgj2j"
Oct 31 22:51:23.830385 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:23.830344    4301 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a93270dfd699138c6bc985c31777bcf8a5ffba0195c49b34e8a2a0e767b4e529"
Oct 31 22:51:32.544133 ip-10-0-135-163 kubenswrapper[4301]: E1031 22:51:32.544090    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 22:51:35.848746 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:35.848715    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b7bc8f456-npf6x"]
Oct 31 22:51:35.849120 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:35.849012    4301 cpu_manager.go:414] "containerMap: removing container" podUID="a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0" containerName="insights-gathering"
Oct 31 22:51:35.849120 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:35.849026    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0" containerName="insights-gathering"
Oct 31 22:51:35.849120 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:35.849076    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0" containerName="insights-gathering"
Oct 31 22:51:35.851102 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:35.851084    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b7bc8f456-npf6x"
Oct 31 22:51:35.864701 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:35.864676    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b7bc8f456-npf6x"]
Oct 31 22:51:35.945138 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:35.945104    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f151329a-9922-4576-8de2-1f8b2034934e-console-oauth-config\") pod \"console-5b7bc8f456-npf6x\" (UID: \"f151329a-9922-4576-8de2-1f8b2034934e\") " pod="openshift-console/console-5b7bc8f456-npf6x"
Oct 31 22:51:35.945138 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:35.945148    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f151329a-9922-4576-8de2-1f8b2034934e-trusted-ca-bundle\") pod \"console-5b7bc8f456-npf6x\" (UID: \"f151329a-9922-4576-8de2-1f8b2034934e\") " pod="openshift-console/console-5b7bc8f456-npf6x"
Oct 31 22:51:35.945414 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:35.945167    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f151329a-9922-4576-8de2-1f8b2034934e-oauth-serving-cert\") pod \"console-5b7bc8f456-npf6x\" (UID: \"f151329a-9922-4576-8de2-1f8b2034934e\") " pod="openshift-console/console-5b7bc8f456-npf6x"
Oct 31 22:51:35.945414 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:35.945306    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f151329a-9922-4576-8de2-1f8b2034934e-console-serving-cert\") pod \"console-5b7bc8f456-npf6x\" (UID: \"f151329a-9922-4576-8de2-1f8b2034934e\") " pod="openshift-console/console-5b7bc8f456-npf6x"
Oct 31 22:51:35.945414 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:35.945338    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f151329a-9922-4576-8de2-1f8b2034934e-service-ca\") pod \"console-5b7bc8f456-npf6x\" (UID: \"f151329a-9922-4576-8de2-1f8b2034934e\") " pod="openshift-console/console-5b7bc8f456-npf6x"
Oct 31 22:51:35.945555 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:35.945424    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f151329a-9922-4576-8de2-1f8b2034934e-tmp\") pod \"console-5b7bc8f456-npf6x\" (UID: \"f151329a-9922-4576-8de2-1f8b2034934e\") " pod="openshift-console/console-5b7bc8f456-npf6x"
Oct 31 22:51:35.945555 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:35.945455    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f151329a-9922-4576-8de2-1f8b2034934e-console-config\") pod \"console-5b7bc8f456-npf6x\" (UID: \"f151329a-9922-4576-8de2-1f8b2034934e\") " pod="openshift-console/console-5b7bc8f456-npf6x"
Oct 31 22:51:35.945555 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:35.945489    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgtlc\" (UniqueName: \"kubernetes.io/projected/f151329a-9922-4576-8de2-1f8b2034934e-kube-api-access-bgtlc\") pod \"console-5b7bc8f456-npf6x\" (UID: \"f151329a-9922-4576-8de2-1f8b2034934e\") " pod="openshift-console/console-5b7bc8f456-npf6x"
Oct 31 22:51:36.046269 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:36.046233    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f151329a-9922-4576-8de2-1f8b2034934e-console-oauth-config\") pod \"console-5b7bc8f456-npf6x\" (UID: \"f151329a-9922-4576-8de2-1f8b2034934e\") " pod="openshift-console/console-5b7bc8f456-npf6x"
Oct 31 22:51:36.046269 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:36.046277    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f151329a-9922-4576-8de2-1f8b2034934e-trusted-ca-bundle\") pod \"console-5b7bc8f456-npf6x\" (UID: \"f151329a-9922-4576-8de2-1f8b2034934e\") " pod="openshift-console/console-5b7bc8f456-npf6x"
Oct 31 22:51:36.046531 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:36.046300    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f151329a-9922-4576-8de2-1f8b2034934e-oauth-serving-cert\") pod \"console-5b7bc8f456-npf6x\" (UID: \"f151329a-9922-4576-8de2-1f8b2034934e\") " pod="openshift-console/console-5b7bc8f456-npf6x"
Oct 31 22:51:36.046531 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:36.046342    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f151329a-9922-4576-8de2-1f8b2034934e-console-serving-cert\") pod \"console-5b7bc8f456-npf6x\" (UID: \"f151329a-9922-4576-8de2-1f8b2034934e\") " pod="openshift-console/console-5b7bc8f456-npf6x"
Oct 31 22:51:36.046531 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:36.046385    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f151329a-9922-4576-8de2-1f8b2034934e-service-ca\") pod \"console-5b7bc8f456-npf6x\" (UID: \"f151329a-9922-4576-8de2-1f8b2034934e\") " pod="openshift-console/console-5b7bc8f456-npf6x"
Oct 31 22:51:36.046531 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:36.046410    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f151329a-9922-4576-8de2-1f8b2034934e-tmp\") pod \"console-5b7bc8f456-npf6x\" (UID: \"f151329a-9922-4576-8de2-1f8b2034934e\") " pod="openshift-console/console-5b7bc8f456-npf6x"
Oct 31 22:51:36.046531 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:36.046440    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f151329a-9922-4576-8de2-1f8b2034934e-console-config\") pod \"console-5b7bc8f456-npf6x\" (UID: \"f151329a-9922-4576-8de2-1f8b2034934e\") " pod="openshift-console/console-5b7bc8f456-npf6x"
Oct 31 22:51:36.046531 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:36.046474    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgtlc\" (UniqueName: \"kubernetes.io/projected/f151329a-9922-4576-8de2-1f8b2034934e-kube-api-access-bgtlc\") pod \"console-5b7bc8f456-npf6x\" (UID: \"f151329a-9922-4576-8de2-1f8b2034934e\") " pod="openshift-console/console-5b7bc8f456-npf6x"
Oct 31 22:51:36.046898 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:36.046862    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f151329a-9922-4576-8de2-1f8b2034934e-tmp\") pod \"console-5b7bc8f456-npf6x\" (UID: \"f151329a-9922-4576-8de2-1f8b2034934e\") " pod="openshift-console/console-5b7bc8f456-npf6x"
Oct 31 22:51:36.047181 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:36.047157    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f151329a-9922-4576-8de2-1f8b2034934e-oauth-serving-cert\") pod \"console-5b7bc8f456-npf6x\" (UID: \"f151329a-9922-4576-8de2-1f8b2034934e\") " pod="openshift-console/console-5b7bc8f456-npf6x"
Oct 31 22:51:36.047277 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:36.047209    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f151329a-9922-4576-8de2-1f8b2034934e-service-ca\") pod \"console-5b7bc8f456-npf6x\" (UID: \"f151329a-9922-4576-8de2-1f8b2034934e\") " pod="openshift-console/console-5b7bc8f456-npf6x"
Oct 31 22:51:36.047277 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:36.047218    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f151329a-9922-4576-8de2-1f8b2034934e-trusted-ca-bundle\") pod \"console-5b7bc8f456-npf6x\" (UID: \"f151329a-9922-4576-8de2-1f8b2034934e\") " pod="openshift-console/console-5b7bc8f456-npf6x"
Oct 31 22:51:36.049143 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:36.047255    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f151329a-9922-4576-8de2-1f8b2034934e-console-config\") pod \"console-5b7bc8f456-npf6x\" (UID: \"f151329a-9922-4576-8de2-1f8b2034934e\") " pod="openshift-console/console-5b7bc8f456-npf6x"
Oct 31 22:51:36.052278 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:36.052250    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f151329a-9922-4576-8de2-1f8b2034934e-console-serving-cert\") pod \"console-5b7bc8f456-npf6x\" (UID: \"f151329a-9922-4576-8de2-1f8b2034934e\") " pod="openshift-console/console-5b7bc8f456-npf6x"
Oct 31 22:51:36.052411 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:36.052389    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f151329a-9922-4576-8de2-1f8b2034934e-console-oauth-config\") pod \"console-5b7bc8f456-npf6x\" (UID: \"f151329a-9922-4576-8de2-1f8b2034934e\") " pod="openshift-console/console-5b7bc8f456-npf6x"
Oct 31 22:51:36.058798 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:36.058776    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgtlc\" (UniqueName: \"kubernetes.io/projected/f151329a-9922-4576-8de2-1f8b2034934e-kube-api-access-bgtlc\") pod \"console-5b7bc8f456-npf6x\" (UID: \"f151329a-9922-4576-8de2-1f8b2034934e\") " pod="openshift-console/console-5b7bc8f456-npf6x"
Oct 31 22:51:36.161227 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:36.161190    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b7bc8f456-npf6x"
Oct 31 22:51:36.434836 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:36.434761    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b7bc8f456-npf6x"]
Oct 31 22:51:36.438405 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:51:36.438379    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf151329a_9922_4576_8de2_1f8b2034934e.slice/crio-769826f701ed00bc1d19abf92b2966fc08049b1e4e22c1a624ddffef4e999be2 WatchSource:0}: Error finding container 769826f701ed00bc1d19abf92b2966fc08049b1e4e22c1a624ddffef4e999be2: Status 404 returned error can't find the container with id 769826f701ed00bc1d19abf92b2966fc08049b1e4e22c1a624ddffef4e999be2
Oct 31 22:51:36.873710 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:36.873674    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b7bc8f456-npf6x" event={"ID":"f151329a-9922-4576-8de2-1f8b2034934e","Type":"ContainerStarted","Data":"f0b1238437836cadb004f9d7c62868cc4833c10df4b89416357364c64603ab3c"}
Oct 31 22:51:36.874086 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:36.873727    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b7bc8f456-npf6x" event={"ID":"f151329a-9922-4576-8de2-1f8b2034934e","Type":"ContainerStarted","Data":"769826f701ed00bc1d19abf92b2966fc08049b1e4e22c1a624ddffef4e999be2"}
Oct 31 22:51:36.896554 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:36.896501    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5b7bc8f456-npf6x" podStartSLOduration=1.893510305 podStartE2EDuration="1.893510305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:51:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:51:36.892775958 +0000 UTC m=+115.639965463" watchObservedRunningTime="2026-10-31 22:51:36.893510305 +0000 UTC m=+115.640699813"
Oct 31 22:51:46.162186 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:46.162152    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b7bc8f456-npf6x"
Oct 31 22:51:46.162594 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:46.162207    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b7bc8f456-npf6x"
Oct 31 22:51:46.166799 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:46.166776    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b7bc8f456-npf6x"
Oct 31 22:51:46.908613 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:46.908584    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b7bc8f456-npf6x"
Oct 31 22:51:46.957819 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:51:46.957790    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85cd978d69-hmq7q"]
Oct 31 22:52:02.544928 ip-10-0-135-163 kubenswrapper[4301]: E1031 22:52:02.544847    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 22:52:11.976936 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:11.976894    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-85cd978d69-hmq7q" podUID="1cd78063-4659-48fc-8937-5db020ee1aa9" containerName="console" containerID="cri-o://3aff5ca63fe2d83d5f87f33656ef716b2505310298774fd885ea3138b4157eba" gracePeriod=15
Oct 31 22:52:12.201635 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:12.201613    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85cd978d69-hmq7q"
Oct 31 22:52:12.288151 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:12.288084    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cd78063-4659-48fc-8937-5db020ee1aa9-tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cd78063-4659-48fc-8937-5db020ee1aa9-tmp\") pod \"1cd78063-4659-48fc-8937-5db020ee1aa9\" (UID: \"1cd78063-4659-48fc-8937-5db020ee1aa9\") "
Oct 31 22:52:12.288151 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:12.288144    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cd78063-4659-48fc-8937-5db020ee1aa9-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1cd78063-4659-48fc-8937-5db020ee1aa9-console-serving-cert\") pod \"1cd78063-4659-48fc-8937-5db020ee1aa9\" (UID: \"1cd78063-4659-48fc-8937-5db020ee1aa9\") "
Oct 31 22:52:12.288403 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:12.288182    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cd78063-4659-48fc-8937-5db020ee1aa9-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1cd78063-4659-48fc-8937-5db020ee1aa9-console-oauth-config\") pod \"1cd78063-4659-48fc-8937-5db020ee1aa9\" (UID: \"1cd78063-4659-48fc-8937-5db020ee1aa9\") "
Oct 31 22:52:12.288403 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:12.288204    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cd78063-4659-48fc-8937-5db020ee1aa9-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1cd78063-4659-48fc-8937-5db020ee1aa9-oauth-serving-cert\") pod \"1cd78063-4659-48fc-8937-5db020ee1aa9\" (UID: \"1cd78063-4659-48fc-8937-5db020ee1aa9\") "
Oct 31 22:52:12.288403 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:12.288233    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cd78063-4659-48fc-8937-5db020ee1aa9-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd78063-4659-48fc-8937-5db020ee1aa9-service-ca\") pod \"1cd78063-4659-48fc-8937-5db020ee1aa9\" (UID: \"1cd78063-4659-48fc-8937-5db020ee1aa9\") "
Oct 31 22:52:12.288403 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:12.288377    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cd78063-4659-48fc-8937-5db020ee1aa9-tmp" pod "1cd78063-4659-48fc-8937-5db020ee1aa9" (UID: "1cd78063-4659-48fc-8937-5db020ee1aa9"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 22:52:12.288615 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:12.288420    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cd78063-4659-48fc-8937-5db020ee1aa9-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cd78063-4659-48fc-8937-5db020ee1aa9-trusted-ca-bundle\") pod \"1cd78063-4659-48fc-8937-5db020ee1aa9\" (UID: \"1cd78063-4659-48fc-8937-5db020ee1aa9\") "
Oct 31 22:52:12.288615 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:12.288463    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cd78063-4659-48fc-8937-5db020ee1aa9-kube-api-access-z28wc\" (UniqueName: \"kubernetes.io/projected/1cd78063-4659-48fc-8937-5db020ee1aa9-kube-api-access-z28wc\") pod \"1cd78063-4659-48fc-8937-5db020ee1aa9\" (UID: \"1cd78063-4659-48fc-8937-5db020ee1aa9\") "
Oct 31 22:52:12.288723 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:12.288667    4301 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cd78063-4659-48fc-8937-5db020ee1aa9-tmp\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 22:52:12.288723 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:12.288664    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd78063-4659-48fc-8937-5db020ee1aa9-service-ca" pod "1cd78063-4659-48fc-8937-5db020ee1aa9" (UID: "1cd78063-4659-48fc-8937-5db020ee1aa9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 22:52:12.288723 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:12.288671    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd78063-4659-48fc-8937-5db020ee1aa9-oauth-serving-cert" pod "1cd78063-4659-48fc-8937-5db020ee1aa9" (UID: "1cd78063-4659-48fc-8937-5db020ee1aa9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 22:52:12.288844 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:12.288764    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd78063-4659-48fc-8937-5db020ee1aa9-trusted-ca-bundle" pod "1cd78063-4659-48fc-8937-5db020ee1aa9" (UID: "1cd78063-4659-48fc-8937-5db020ee1aa9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 22:52:12.290438 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:12.290410    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cd78063-4659-48fc-8937-5db020ee1aa9-console-oauth-config" pod "1cd78063-4659-48fc-8937-5db020ee1aa9" (UID: "1cd78063-4659-48fc-8937-5db020ee1aa9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 22:52:12.290533 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:12.290443    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd78063-4659-48fc-8937-5db020ee1aa9-kube-api-access-z28wc" pod "1cd78063-4659-48fc-8937-5db020ee1aa9" (UID: "1cd78063-4659-48fc-8937-5db020ee1aa9"). InnerVolumeSpecName "kube-api-access-z28wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 22:52:12.290613 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:12.290598    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cd78063-4659-48fc-8937-5db020ee1aa9-console-serving-cert" pod "1cd78063-4659-48fc-8937-5db020ee1aa9" (UID: "1cd78063-4659-48fc-8937-5db020ee1aa9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 22:52:12.389240 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:12.389187    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cd78063-4659-48fc-8937-5db020ee1aa9-console-config\" (UniqueName: \"kubernetes.io/configmap/1cd78063-4659-48fc-8937-5db020ee1aa9-console-config\") pod \"1cd78063-4659-48fc-8937-5db020ee1aa9\" (UID: \"1cd78063-4659-48fc-8937-5db020ee1aa9\") "
Oct 31 22:52:12.389432 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:12.389323    4301 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1cd78063-4659-48fc-8937-5db020ee1aa9-console-serving-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 22:52:12.389432 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:12.389336    4301 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1cd78063-4659-48fc-8937-5db020ee1aa9-console-oauth-config\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 22:52:12.389432 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:12.389346    4301 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1cd78063-4659-48fc-8937-5db020ee1aa9-oauth-serving-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 22:52:12.389432 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:12.389373    4301 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd78063-4659-48fc-8937-5db020ee1aa9-service-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 22:52:12.389432 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:12.389382    4301 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cd78063-4659-48fc-8937-5db020ee1aa9-trusted-ca-bundle\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 22:52:12.389432 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:12.389391    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z28wc\" (UniqueName: \"kubernetes.io/projected/1cd78063-4659-48fc-8937-5db020ee1aa9-kube-api-access-z28wc\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 22:52:12.389628 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:12.389540    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd78063-4659-48fc-8937-5db020ee1aa9-console-config" pod "1cd78063-4659-48fc-8937-5db020ee1aa9" (UID: "1cd78063-4659-48fc-8937-5db020ee1aa9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 22:52:12.489683 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:12.489655    4301 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1cd78063-4659-48fc-8937-5db020ee1aa9-console-config\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 22:52:12.994996 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:12.994961    4301 generic.go:356] "Generic (PLEG): container finished" podID="1cd78063-4659-48fc-8937-5db020ee1aa9" containerID="3aff5ca63fe2d83d5f87f33656ef716b2505310298774fd885ea3138b4157eba" exitCode=0
Oct 31 22:52:12.995410 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:12.995012    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85cd978d69-hmq7q"
Oct 31 22:52:12.995410 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:12.995044    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85cd978d69-hmq7q" event={"ID":"1cd78063-4659-48fc-8937-5db020ee1aa9","Type":"ContainerDied","Data":"3aff5ca63fe2d83d5f87f33656ef716b2505310298774fd885ea3138b4157eba"}
Oct 31 22:52:12.995410 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:12.995095    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85cd978d69-hmq7q" event={"ID":"1cd78063-4659-48fc-8937-5db020ee1aa9","Type":"ContainerDied","Data":"813503e535873ee1ab4c931193d36047f47eb6c6c12e41831d54e33e577cf663"}
Oct 31 22:52:12.995410 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:12.995115    4301 scope.go:122] "RemoveContainer" containerID="3aff5ca63fe2d83d5f87f33656ef716b2505310298774fd885ea3138b4157eba"
Oct 31 22:52:13.003506 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:13.003488    4301 scope.go:122] "RemoveContainer" containerID="3aff5ca63fe2d83d5f87f33656ef716b2505310298774fd885ea3138b4157eba"
Oct 31 22:52:13.003796 ip-10-0-135-163 kubenswrapper[4301]: E1031 22:52:13.003770    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aff5ca63fe2d83d5f87f33656ef716b2505310298774fd885ea3138b4157eba\": container with ID starting with 3aff5ca63fe2d83d5f87f33656ef716b2505310298774fd885ea3138b4157eba not found: ID does not exist" containerID="3aff5ca63fe2d83d5f87f33656ef716b2505310298774fd885ea3138b4157eba"
Oct 31 22:52:13.003872 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:13.003801    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aff5ca63fe2d83d5f87f33656ef716b2505310298774fd885ea3138b4157eba"} err="failed to get container status \"3aff5ca63fe2d83d5f87f33656ef716b2505310298774fd885ea3138b4157eba\": rpc error: code = NotFound desc = could not find container \"3aff5ca63fe2d83d5f87f33656ef716b2505310298774fd885ea3138b4157eba\": container with ID starting with 3aff5ca63fe2d83d5f87f33656ef716b2505310298774fd885ea3138b4157eba not found: ID does not exist"
Oct 31 22:52:13.015435 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:13.015413    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85cd978d69-hmq7q"]
Oct 31 22:52:13.020817 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:13.020797    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85cd978d69-hmq7q"]
Oct 31 22:52:13.400415 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:52:13.400383    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cd78063-4659-48fc-8937-5db020ee1aa9" path="/var/lib/kubelet/pods/1cd78063-4659-48fc-8937-5db020ee1aa9/volumes"
Oct 31 22:52:32.544060 ip-10-0-135-163 kubenswrapper[4301]: E1031 22:52:32.544022    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 22:53:02.543701 ip-10-0-135-163 kubenswrapper[4301]: E1031 22:53:02.543666    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 22:53:09.782332 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:09.782299    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mj2v"]
Oct 31 22:53:09.782950 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:09.782928    4301 cpu_manager.go:414] "containerMap: removing container" podUID="1cd78063-4659-48fc-8937-5db020ee1aa9" containerName="console"
Oct 31 22:53:09.783028 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:09.782950    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd78063-4659-48fc-8937-5db020ee1aa9" containerName="console"
Oct 31 22:53:09.783081 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:09.783049    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cd78063-4659-48fc-8937-5db020ee1aa9" containerName="console"
Oct 31 22:53:09.784940 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:09.784920    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mj2v"
Oct 31 22:53:09.795589 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:09.795566    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-h6jfg\""
Oct 31 22:53:09.795709 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:09.795687    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 31 22:53:09.795759 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:09.795726    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 31 22:53:09.802656 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:09.802634    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mj2v"]
Oct 31 22:53:09.876258 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:09.876220    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bbtv4"]
Oct 31 22:53:09.878204 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:09.878187    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bbtv4"
Oct 31 22:53:09.889206 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:09.889186    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/236ef1f5-169f-4c41-b8c5-d2889b448621-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mj2v\" (UID: \"236ef1f5-169f-4c41-b8c5-d2889b448621\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mj2v"
Oct 31 22:53:09.889310 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:09.889215    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcr7h\" (UniqueName: \"kubernetes.io/projected/236ef1f5-169f-4c41-b8c5-d2889b448621-kube-api-access-lcr7h\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mj2v\" (UID: \"236ef1f5-169f-4c41-b8c5-d2889b448621\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mj2v"
Oct 31 22:53:09.889310 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:09.889296    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/236ef1f5-169f-4c41-b8c5-d2889b448621-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mj2v\" (UID: \"236ef1f5-169f-4c41-b8c5-d2889b448621\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mj2v"
Oct 31 22:53:09.893547 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:09.893524    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bbtv4"]
Oct 31 22:53:09.903627 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:09.903607    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-ldmfr\""
Oct 31 22:53:09.989959 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:09.989924    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/236ef1f5-169f-4c41-b8c5-d2889b448621-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mj2v\" (UID: \"236ef1f5-169f-4c41-b8c5-d2889b448621\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mj2v"
Oct 31 22:53:09.990125 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:09.989966    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cfe8af1c-3c89-492e-ad64-bd40c46361b8-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bbtv4\" (UID: \"cfe8af1c-3c89-492e-ad64-bd40c46361b8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bbtv4"
Oct 31 22:53:09.990125 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:09.989991    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxgzw\" (UniqueName: \"kubernetes.io/projected/cfe8af1c-3c89-492e-ad64-bd40c46361b8-kube-api-access-hxgzw\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bbtv4\" (UID: \"cfe8af1c-3c89-492e-ad64-bd40c46361b8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bbtv4"
Oct 31 22:53:09.990208 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:09.990134    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/236ef1f5-169f-4c41-b8c5-d2889b448621-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mj2v\" (UID: \"236ef1f5-169f-4c41-b8c5-d2889b448621\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mj2v"
Oct 31 22:53:09.990208 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:09.990164    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcr7h\" (UniqueName: \"kubernetes.io/projected/236ef1f5-169f-4c41-b8c5-d2889b448621-kube-api-access-lcr7h\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mj2v\" (UID: \"236ef1f5-169f-4c41-b8c5-d2889b448621\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mj2v"
Oct 31 22:53:09.990208 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:09.990185    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cfe8af1c-3c89-492e-ad64-bd40c46361b8-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bbtv4\" (UID: \"cfe8af1c-3c89-492e-ad64-bd40c46361b8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bbtv4"
Oct 31 22:53:09.990346 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:09.990325    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/236ef1f5-169f-4c41-b8c5-d2889b448621-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mj2v\" (UID: \"236ef1f5-169f-4c41-b8c5-d2889b448621\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mj2v"
Oct 31 22:53:09.990417 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:09.990407    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/236ef1f5-169f-4c41-b8c5-d2889b448621-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mj2v\" (UID: \"236ef1f5-169f-4c41-b8c5-d2889b448621\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mj2v"
Oct 31 22:53:10.002071 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.002048    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcr7h\" (UniqueName: \"kubernetes.io/projected/236ef1f5-169f-4c41-b8c5-d2889b448621-kube-api-access-lcr7h\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mj2v\" (UID: \"236ef1f5-169f-4c41-b8c5-d2889b448621\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mj2v"
Oct 31 22:53:10.091150 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.091072    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cfe8af1c-3c89-492e-ad64-bd40c46361b8-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bbtv4\" (UID: \"cfe8af1c-3c89-492e-ad64-bd40c46361b8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bbtv4"
Oct 31 22:53:10.091150 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.091117    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cfe8af1c-3c89-492e-ad64-bd40c46361b8-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bbtv4\" (UID: \"cfe8af1c-3c89-492e-ad64-bd40c46361b8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bbtv4"
Oct 31 22:53:10.091150 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.091141    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxgzw\" (UniqueName: \"kubernetes.io/projected/cfe8af1c-3c89-492e-ad64-bd40c46361b8-kube-api-access-hxgzw\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bbtv4\" (UID: \"cfe8af1c-3c89-492e-ad64-bd40c46361b8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bbtv4"
Oct 31 22:53:10.091551 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.091532    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cfe8af1c-3c89-492e-ad64-bd40c46361b8-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bbtv4\" (UID: \"cfe8af1c-3c89-492e-ad64-bd40c46361b8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bbtv4"
Oct 31 22:53:10.091627 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.091557    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cfe8af1c-3c89-492e-ad64-bd40c46361b8-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bbtv4\" (UID: \"cfe8af1c-3c89-492e-ad64-bd40c46361b8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bbtv4"
Oct 31 22:53:10.100808 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.100787    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mj2v"
Oct 31 22:53:10.102901 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.102838    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxgzw\" (UniqueName: \"kubernetes.io/projected/cfe8af1c-3c89-492e-ad64-bd40c46361b8-kube-api-access-hxgzw\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bbtv4\" (UID: \"cfe8af1c-3c89-492e-ad64-bd40c46361b8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bbtv4"
Oct 31 22:53:10.189678 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.189644    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bbtv4"
Oct 31 22:53:10.253840 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.253811    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565"]
Oct 31 22:53:10.256622 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.256590    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565"
Oct 31 22:53:10.260557 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.260532    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 22:53:10.260705 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.260641    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 22:53:10.261045 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.261028    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 22:53:10.262216 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.262192    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 22:53:10.262216 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.262192    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 22:53:10.262350 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.262308    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 22:53:10.262350 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.262336    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-v7rzj\""
Oct 31 22:53:10.270412 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.270393    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565"]
Oct 31 22:53:10.371107 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.371026    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mj2v"]
Oct 31 22:53:10.375254 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:53:10.375230    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod236ef1f5_169f_4c41_b8c5_d2889b448621.slice/crio-0ae301e733eab4717a5d8d485b7a0cf996c8a63e0b0bc6b4228fbdbd97060d0b WatchSource:0}: Error finding container 0ae301e733eab4717a5d8d485b7a0cf996c8a63e0b0bc6b4228fbdbd97060d0b: Status 404 returned error can't find the container with id 0ae301e733eab4717a5d8d485b7a0cf996c8a63e0b0bc6b4228fbdbd97060d0b
Oct 31 22:53:10.392964 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.392943    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-k6565\" (UID: \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565"
Oct 31 22:53:10.393078 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.392973    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-k6565\" (UID: \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565"
Oct 31 22:53:10.393078 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.392994    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-k6565\" (UID: \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565"
Oct 31 22:53:10.393078 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.393053    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-k6565\" (UID: \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565"
Oct 31 22:53:10.393186 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.393124    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-k6565\" (UID: \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565"
Oct 31 22:53:10.393186 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.393177    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-k6565\" (UID: \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565"
Oct 31 22:53:10.393259 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.393196    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlhq7\" (UniqueName: \"kubernetes.io/projected/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-kube-api-access-mlhq7\") pod \"istiod-openshift-gateway-7876bfcb74-k6565\" (UID: \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565"
Oct 31 22:53:10.460625 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.460594    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bbtv4"]
Oct 31 22:53:10.464160 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:53:10.464129    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfe8af1c_3c89_492e_ad64_bd40c46361b8.slice/crio-a4596665b7fcfb6e4e38856ca6c13a9f7e512f52337db382910dcc2842714e1f WatchSource:0}: Error finding container a4596665b7fcfb6e4e38856ca6c13a9f7e512f52337db382910dcc2842714e1f: Status 404 returned error can't find the container with id a4596665b7fcfb6e4e38856ca6c13a9f7e512f52337db382910dcc2842714e1f
Oct 31 22:53:10.494404 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.494381    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-k6565\" (UID: \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565"
Oct 31 22:53:10.494580 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.494413    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-k6565\" (UID: \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565"
Oct 31 22:53:10.494580 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.494430    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlhq7\" (UniqueName: \"kubernetes.io/projected/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-kube-api-access-mlhq7\") pod \"istiod-openshift-gateway-7876bfcb74-k6565\" (UID: \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565"
Oct 31 22:53:10.494580 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.494478    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-k6565\" (UID: \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565"
Oct 31 22:53:10.494580 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.494498    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-k6565\" (UID: \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565"
Oct 31 22:53:10.494580 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.494513    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-k6565\" (UID: \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565"
Oct 31 22:53:10.494580 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.494535    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-k6565\" (UID: \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565"
Oct 31 22:53:10.495164 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.495139    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-k6565\" (UID: \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565"
Oct 31 22:53:10.496733 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.496705    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-k6565\" (UID: \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565"
Oct 31 22:53:10.496832 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.496755    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-k6565\" (UID: \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565"
Oct 31 22:53:10.496924 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.496905    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-k6565\" (UID: \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565"
Oct 31 22:53:10.496979 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.496930    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-k6565\" (UID: \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565"
Oct 31 22:53:10.505969 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.505944    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-k6565\" (UID: \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565"
Oct 31 22:53:10.506177 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.506162    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlhq7\" (UniqueName: \"kubernetes.io/projected/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-kube-api-access-mlhq7\") pod \"istiod-openshift-gateway-7876bfcb74-k6565\" (UID: \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565"
Oct 31 22:53:10.568177 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.568147    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565"
Oct 31 22:53:10.843544 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:10.843463    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565"]
Oct 31 22:53:10.846843 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:53:10.846817    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf733ecfd_bd0b_41d4_b4b1_d6b929bfb6c2.slice/crio-b7c2ec876a66ecf8bf76bb24a1486434ce54d581b58c59fd5229c89265c03d08 WatchSource:0}: Error finding container b7c2ec876a66ecf8bf76bb24a1486434ce54d581b58c59fd5229c89265c03d08: Status 404 returned error can't find the container with id b7c2ec876a66ecf8bf76bb24a1486434ce54d581b58c59fd5229c89265c03d08
Oct 31 22:53:11.180727 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:11.180650    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565" event={"ID":"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2","Type":"ContainerStarted","Data":"b7c2ec876a66ecf8bf76bb24a1486434ce54d581b58c59fd5229c89265c03d08"}
Oct 31 22:53:11.182100 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:11.182037    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bbtv4" event={"ID":"cfe8af1c-3c89-492e-ad64-bd40c46361b8","Type":"ContainerStarted","Data":"a4596665b7fcfb6e4e38856ca6c13a9f7e512f52337db382910dcc2842714e1f"}
Oct 31 22:53:11.183633 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:11.183602    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mj2v" event={"ID":"236ef1f5-169f-4c41-b8c5-d2889b448621","Type":"ContainerStarted","Data":"0ae301e733eab4717a5d8d485b7a0cf996c8a63e0b0bc6b4228fbdbd97060d0b"}
Oct 31 22:53:14.614149 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:14.614104    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 22:53:14.614595 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:14.614192    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 22:53:15.206313 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:15.205729    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565" event={"ID":"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2","Type":"ContainerStarted","Data":"d8bed165fc00064230a86de05b6d9fd2dba5edb6810255792a2b35793a441ce2"}
Oct 31 22:53:15.206313 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:15.205989    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565"
Oct 31 22:53:15.208314 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:15.208265    4301 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-k6565 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 22:53:15.208462 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:15.208338    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565" podUID="f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 22:53:16.211240 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:16.211206    4301 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-k6565 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 22:53:16.211675 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:16.211266    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565" podUID="f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 22:53:17.216348 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:17.216315    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bbtv4" event={"ID":"cfe8af1c-3c89-492e-ad64-bd40c46361b8","Type":"ContainerStarted","Data":"fb9ac33abba146d4d8b02346b29ff8584b51d51dc6514b6f1a5dabf708ba600d"}
Oct 31 22:53:17.237436 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:17.237379    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565" podStartSLOduration=3.4730115919999998 podStartE2EDuration="7.237343228s" totalImagesPullingTime="3.764331636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:53:15.229104176 +0000 UTC m=+213.976293694" watchObservedRunningTime="2026-10-31 22:53:17.237343228 +0000 UTC m=+215.984532735"
Oct 31 22:53:18.221427 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:18.221389    4301 generic.go:356] "Generic (PLEG): container finished" podID="236ef1f5-169f-4c41-b8c5-d2889b448621" containerID="7b649873906e7c8c3d19bfe9a0897f9aefcf15a7524632a758afafbae784eeb8" exitCode=0
Oct 31 22:53:18.221851 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:18.221540    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mj2v" event={"ID":"236ef1f5-169f-4c41-b8c5-d2889b448621","Type":"ContainerDied","Data":"7b649873906e7c8c3d19bfe9a0897f9aefcf15a7524632a758afafbae784eeb8"}
Oct 31 22:53:18.227797 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:18.227770    4301 generic.go:356] "Generic (PLEG): container finished" podID="cfe8af1c-3c89-492e-ad64-bd40c46361b8" containerID="fb9ac33abba146d4d8b02346b29ff8584b51d51dc6514b6f1a5dabf708ba600d" exitCode=0
Oct 31 22:53:18.228046 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:18.227804    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bbtv4" event={"ID":"cfe8af1c-3c89-492e-ad64-bd40c46361b8","Type":"ContainerDied","Data":"fb9ac33abba146d4d8b02346b29ff8584b51d51dc6514b6f1a5dabf708ba600d"}
Oct 31 22:53:19.211236 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:19.211147    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565"
Oct 31 22:53:19.233754 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:19.233719    4301 generic.go:356] "Generic (PLEG): container finished" podID="cfe8af1c-3c89-492e-ad64-bd40c46361b8" containerID="c9006fb373825da52a38eb4f6350e04514f18355964f9064ffdd539c92236334" exitCode=0
Oct 31 22:53:19.234441 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:19.233794    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bbtv4" event={"ID":"cfe8af1c-3c89-492e-ad64-bd40c46361b8","Type":"ContainerDied","Data":"c9006fb373825da52a38eb4f6350e04514f18355964f9064ffdd539c92236334"}
Oct 31 22:53:21.244024 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:21.243979    4301 generic.go:356] "Generic (PLEG): container finished" podID="236ef1f5-169f-4c41-b8c5-d2889b448621" containerID="6c57bc98cc48bf40cd0f212a53f5970ebf93fadac8d2f8645823dc1dbe205917" exitCode=0
Oct 31 22:53:21.244554 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:21.244021    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mj2v" event={"ID":"236ef1f5-169f-4c41-b8c5-d2889b448621","Type":"ContainerDied","Data":"6c57bc98cc48bf40cd0f212a53f5970ebf93fadac8d2f8645823dc1dbe205917"}
Oct 31 22:53:26.266337 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:26.266304    4301 generic.go:356] "Generic (PLEG): container finished" podID="cfe8af1c-3c89-492e-ad64-bd40c46361b8" containerID="018708b90053bb47e38ae8463422a9a660dd5d827c01051eafef78f0453984be" exitCode=0
Oct 31 22:53:26.266761 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:26.266423    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bbtv4" event={"ID":"cfe8af1c-3c89-492e-ad64-bd40c46361b8","Type":"ContainerDied","Data":"018708b90053bb47e38ae8463422a9a660dd5d827c01051eafef78f0453984be"}
Oct 31 22:53:27.411315 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:27.411285    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bbtv4"
Oct 31 22:53:27.447430 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:27.447376    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe8af1c-3c89-492e-ad64-bd40c46361b8-util\" (UniqueName: \"kubernetes.io/empty-dir/cfe8af1c-3c89-492e-ad64-bd40c46361b8-util\") pod \"cfe8af1c-3c89-492e-ad64-bd40c46361b8\" (UID: \"cfe8af1c-3c89-492e-ad64-bd40c46361b8\") "
Oct 31 22:53:27.447598 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:27.447474    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe8af1c-3c89-492e-ad64-bd40c46361b8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/cfe8af1c-3c89-492e-ad64-bd40c46361b8-bundle\") pod \"cfe8af1c-3c89-492e-ad64-bd40c46361b8\" (UID: \"cfe8af1c-3c89-492e-ad64-bd40c46361b8\") "
Oct 31 22:53:27.447598 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:27.447519    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfe8af1c-3c89-492e-ad64-bd40c46361b8-kube-api-access-hxgzw\" (UniqueName: \"kubernetes.io/projected/cfe8af1c-3c89-492e-ad64-bd40c46361b8-kube-api-access-hxgzw\") pod \"cfe8af1c-3c89-492e-ad64-bd40c46361b8\" (UID: \"cfe8af1c-3c89-492e-ad64-bd40c46361b8\") "
Oct 31 22:53:27.449531 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:27.449481    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe8af1c-3c89-492e-ad64-bd40c46361b8-bundle" pod "cfe8af1c-3c89-492e-ad64-bd40c46361b8" (UID: "cfe8af1c-3c89-492e-ad64-bd40c46361b8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 22:53:27.450271 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:27.450239    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe8af1c-3c89-492e-ad64-bd40c46361b8-kube-api-access-hxgzw" pod "cfe8af1c-3c89-492e-ad64-bd40c46361b8" (UID: "cfe8af1c-3c89-492e-ad64-bd40c46361b8"). InnerVolumeSpecName "kube-api-access-hxgzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 22:53:27.454129 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:27.454098    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe8af1c-3c89-492e-ad64-bd40c46361b8-util" pod "cfe8af1c-3c89-492e-ad64-bd40c46361b8" (UID: "cfe8af1c-3c89-492e-ad64-bd40c46361b8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 22:53:27.548798 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:27.548715    4301 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cfe8af1c-3c89-492e-ad64-bd40c46361b8-util\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 22:53:27.548798 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:27.548757    4301 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cfe8af1c-3c89-492e-ad64-bd40c46361b8-bundle\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 22:53:27.548798 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:27.548773    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxgzw\" (UniqueName: \"kubernetes.io/projected/cfe8af1c-3c89-492e-ad64-bd40c46361b8-kube-api-access-hxgzw\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 22:53:28.275133 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:28.275095    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bbtv4" event={"ID":"cfe8af1c-3c89-492e-ad64-bd40c46361b8","Type":"ContainerDied","Data":"a4596665b7fcfb6e4e38856ca6c13a9f7e512f52337db382910dcc2842714e1f"}
Oct 31 22:53:28.275133 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:28.275122    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bbtv4"
Oct 31 22:53:28.275349 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:28.275149    4301 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4596665b7fcfb6e4e38856ca6c13a9f7e512f52337db382910dcc2842714e1f"
Oct 31 22:53:28.277427 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:28.277401    4301 generic.go:356] "Generic (PLEG): container finished" podID="236ef1f5-169f-4c41-b8c5-d2889b448621" containerID="c4438632ff47a2d2d8cb28341077380b07875adc8397827eb17451a65fd0bd71" exitCode=0
Oct 31 22:53:28.277544 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:28.277464    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mj2v" event={"ID":"236ef1f5-169f-4c41-b8c5-d2889b448621","Type":"ContainerDied","Data":"c4438632ff47a2d2d8cb28341077380b07875adc8397827eb17451a65fd0bd71"}
Oct 31 22:53:29.399668 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:29.399642    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mj2v"
Oct 31 22:53:29.461815 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:29.461787    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/236ef1f5-169f-4c41-b8c5-d2889b448621-kube-api-access-lcr7h\" (UniqueName: \"kubernetes.io/projected/236ef1f5-169f-4c41-b8c5-d2889b448621-kube-api-access-lcr7h\") pod \"236ef1f5-169f-4c41-b8c5-d2889b448621\" (UID: \"236ef1f5-169f-4c41-b8c5-d2889b448621\") "
Oct 31 22:53:29.461952 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:29.461832    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/236ef1f5-169f-4c41-b8c5-d2889b448621-bundle\" (UniqueName: \"kubernetes.io/empty-dir/236ef1f5-169f-4c41-b8c5-d2889b448621-bundle\") pod \"236ef1f5-169f-4c41-b8c5-d2889b448621\" (UID: \"236ef1f5-169f-4c41-b8c5-d2889b448621\") "
Oct 31 22:53:29.461952 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:29.461852    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/236ef1f5-169f-4c41-b8c5-d2889b448621-util\" (UniqueName: \"kubernetes.io/empty-dir/236ef1f5-169f-4c41-b8c5-d2889b448621-util\") pod \"236ef1f5-169f-4c41-b8c5-d2889b448621\" (UID: \"236ef1f5-169f-4c41-b8c5-d2889b448621\") "
Oct 31 22:53:29.462439 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:29.462417    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/236ef1f5-169f-4c41-b8c5-d2889b448621-bundle" pod "236ef1f5-169f-4c41-b8c5-d2889b448621" (UID: "236ef1f5-169f-4c41-b8c5-d2889b448621"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 22:53:29.463827 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:29.463802    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/236ef1f5-169f-4c41-b8c5-d2889b448621-kube-api-access-lcr7h" pod "236ef1f5-169f-4c41-b8c5-d2889b448621" (UID: "236ef1f5-169f-4c41-b8c5-d2889b448621"). InnerVolumeSpecName "kube-api-access-lcr7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 22:53:29.466117 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:29.466097    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/236ef1f5-169f-4c41-b8c5-d2889b448621-util" pod "236ef1f5-169f-4c41-b8c5-d2889b448621" (UID: "236ef1f5-169f-4c41-b8c5-d2889b448621"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 22:53:29.562430 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:29.562331    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcr7h\" (UniqueName: \"kubernetes.io/projected/236ef1f5-169f-4c41-b8c5-d2889b448621-kube-api-access-lcr7h\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 22:53:29.562430 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:29.562385    4301 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/236ef1f5-169f-4c41-b8c5-d2889b448621-bundle\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 22:53:29.562430 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:29.562395    4301 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/236ef1f5-169f-4c41-b8c5-d2889b448621-util\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 22:53:30.286571 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:30.286534    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mj2v" event={"ID":"236ef1f5-169f-4c41-b8c5-d2889b448621","Type":"ContainerDied","Data":"0ae301e733eab4717a5d8d485b7a0cf996c8a63e0b0bc6b4228fbdbd97060d0b"}
Oct 31 22:53:30.286571 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:30.286574    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mj2v"
Oct 31 22:53:30.286779 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:30.286580    4301 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ae301e733eab4717a5d8d485b7a0cf996c8a63e0b0bc6b4228fbdbd97060d0b"
Oct 31 22:53:32.543534 ip-10-0-135-163 kubenswrapper[4301]: E1031 22:53:32.543496    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 22:53:38.538698 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.538659    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q"]
Oct 31 22:53:38.539169 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.538901    4301 cpu_manager.go:414] "containerMap: removing container" podUID="236ef1f5-169f-4c41-b8c5-d2889b448621" containerName="pull"
Oct 31 22:53:38.539169 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.538911    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="236ef1f5-169f-4c41-b8c5-d2889b448621" containerName="pull"
Oct 31 22:53:38.539169 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.538925    4301 cpu_manager.go:414] "containerMap: removing container" podUID="236ef1f5-169f-4c41-b8c5-d2889b448621" containerName="util"
Oct 31 22:53:38.539169 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.538929    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="236ef1f5-169f-4c41-b8c5-d2889b448621" containerName="util"
Oct 31 22:53:38.539169 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.538938    4301 cpu_manager.go:414] "containerMap: removing container" podUID="cfe8af1c-3c89-492e-ad64-bd40c46361b8" containerName="pull"
Oct 31 22:53:38.539169 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.538943    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfe8af1c-3c89-492e-ad64-bd40c46361b8" containerName="pull"
Oct 31 22:53:38.539169 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.538950    4301 cpu_manager.go:414] "containerMap: removing container" podUID="cfe8af1c-3c89-492e-ad64-bd40c46361b8" containerName="extract"
Oct 31 22:53:38.539169 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.538954    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfe8af1c-3c89-492e-ad64-bd40c46361b8" containerName="extract"
Oct 31 22:53:38.539169 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.538961    4301 cpu_manager.go:414] "containerMap: removing container" podUID="236ef1f5-169f-4c41-b8c5-d2889b448621" containerName="extract"
Oct 31 22:53:38.539169 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.538965    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="236ef1f5-169f-4c41-b8c5-d2889b448621" containerName="extract"
Oct 31 22:53:38.539169 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.538975    4301 cpu_manager.go:414] "containerMap: removing container" podUID="cfe8af1c-3c89-492e-ad64-bd40c46361b8" containerName="util"
Oct 31 22:53:38.539169 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.538979    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfe8af1c-3c89-492e-ad64-bd40c46361b8" containerName="util"
Oct 31 22:53:38.539169 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.539027    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfe8af1c-3c89-492e-ad64-bd40c46361b8" containerName="extract"
Oct 31 22:53:38.539169 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.539034    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="236ef1f5-169f-4c41-b8c5-d2889b448621" containerName="extract"
Oct 31 22:53:38.542824 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.542806    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q"
Oct 31 22:53:38.581217 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.581189    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q"]
Oct 31 22:53:38.626215 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.626185    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 31 22:53:38.626215 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.626216    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 31 22:53:38.627547 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.627521    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 31 22:53:38.627547 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.627522    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 31 22:53:38.627694 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.627587    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 31 22:53:38.627694 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.627640    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-hg7q6\""
Oct 31 22:53:38.630502 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.630484    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/19ea0f64-10fa-438a-aa25-724be70d152f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q\" (UID: \"19ea0f64-10fa-438a-aa25-724be70d152f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q"
Oct 31 22:53:38.630574 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.630512    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19ea0f64-10fa-438a-aa25-724be70d152f-tmp\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q\" (UID: \"19ea0f64-10fa-438a-aa25-724be70d152f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q"
Oct 31 22:53:38.630574 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.630546    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6z8d\" (UniqueName: \"kubernetes.io/projected/19ea0f64-10fa-438a-aa25-724be70d152f-kube-api-access-t6z8d\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q\" (UID: \"19ea0f64-10fa-438a-aa25-724be70d152f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q"
Oct 31 22:53:38.630669 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.630654    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19ea0f64-10fa-438a-aa25-724be70d152f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q\" (UID: \"19ea0f64-10fa-438a-aa25-724be70d152f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q"
Oct 31 22:53:38.630711 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.630675    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/19ea0f64-10fa-438a-aa25-724be70d152f-manifests\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q\" (UID: \"19ea0f64-10fa-438a-aa25-724be70d152f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q"
Oct 31 22:53:38.630762 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.630732    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/19ea0f64-10fa-438a-aa25-724be70d152f-metrics-cert\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q\" (UID: \"19ea0f64-10fa-438a-aa25-724be70d152f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q"
Oct 31 22:53:38.731456 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.731425    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6z8d\" (UniqueName: \"kubernetes.io/projected/19ea0f64-10fa-438a-aa25-724be70d152f-kube-api-access-t6z8d\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q\" (UID: \"19ea0f64-10fa-438a-aa25-724be70d152f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q"
Oct 31 22:53:38.731456 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.731458    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19ea0f64-10fa-438a-aa25-724be70d152f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q\" (UID: \"19ea0f64-10fa-438a-aa25-724be70d152f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q"
Oct 31 22:53:38.731704 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.731478    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/19ea0f64-10fa-438a-aa25-724be70d152f-manifests\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q\" (UID: \"19ea0f64-10fa-438a-aa25-724be70d152f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q"
Oct 31 22:53:38.731704 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.731500    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/19ea0f64-10fa-438a-aa25-724be70d152f-metrics-cert\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q\" (UID: \"19ea0f64-10fa-438a-aa25-724be70d152f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q"
Oct 31 22:53:38.731704 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.731529    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/19ea0f64-10fa-438a-aa25-724be70d152f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q\" (UID: \"19ea0f64-10fa-438a-aa25-724be70d152f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q"
Oct 31 22:53:38.731704 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.731561    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19ea0f64-10fa-438a-aa25-724be70d152f-tmp\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q\" (UID: \"19ea0f64-10fa-438a-aa25-724be70d152f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q"
Oct 31 22:53:38.731965 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.731935    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/19ea0f64-10fa-438a-aa25-724be70d152f-manifests\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q\" (UID: \"19ea0f64-10fa-438a-aa25-724be70d152f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q"
Oct 31 22:53:38.734010 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.733983    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/19ea0f64-10fa-438a-aa25-724be70d152f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q\" (UID: \"19ea0f64-10fa-438a-aa25-724be70d152f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q"
Oct 31 22:53:38.734126 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.734031    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/19ea0f64-10fa-438a-aa25-724be70d152f-metrics-cert\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q\" (UID: \"19ea0f64-10fa-438a-aa25-724be70d152f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q"
Oct 31 22:53:38.734410 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.734390    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19ea0f64-10fa-438a-aa25-724be70d152f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q\" (UID: \"19ea0f64-10fa-438a-aa25-724be70d152f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q"
Oct 31 22:53:38.734520 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.734503    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19ea0f64-10fa-438a-aa25-724be70d152f-tmp\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q\" (UID: \"19ea0f64-10fa-438a-aa25-724be70d152f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q"
Oct 31 22:53:38.748582 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.748557    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6z8d\" (UniqueName: \"kubernetes.io/projected/19ea0f64-10fa-438a-aa25-724be70d152f-kube-api-access-t6z8d\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q\" (UID: \"19ea0f64-10fa-438a-aa25-724be70d152f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q"
Oct 31 22:53:38.855205 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:38.855132    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q"
Oct 31 22:53:39.157941 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:39.157908    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q"]
Oct 31 22:53:39.163308 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:53:39.163280    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ea0f64_10fa_438a_aa25_724be70d152f.slice/crio-c882e6276760a169b1142a2fe900c2f8af97f976d43f5038135a0f2541baf987 WatchSource:0}: Error finding container c882e6276760a169b1142a2fe900c2f8af97f976d43f5038135a0f2541baf987: Status 404 returned error can't find the container with id c882e6276760a169b1142a2fe900c2f8af97f976d43f5038135a0f2541baf987
Oct 31 22:53:39.317326 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:39.317294    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q" event={"ID":"19ea0f64-10fa-438a-aa25-724be70d152f","Type":"ContainerStarted","Data":"c882e6276760a169b1142a2fe900c2f8af97f976d43f5038135a0f2541baf987"}
Oct 31 22:53:42.335550 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:42.335458    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q" event={"ID":"19ea0f64-10fa-438a-aa25-724be70d152f","Type":"ContainerStarted","Data":"fe0ddd63abe231ef665ac0a12652a39fa4c8dae5feaa942dee60bd90ce217c63"}
Oct 31 22:53:43.340630 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:43.340592    4301 generic.go:356] "Generic (PLEG): container finished" podID="19ea0f64-10fa-438a-aa25-724be70d152f" containerID="fe0ddd63abe231ef665ac0a12652a39fa4c8dae5feaa942dee60bd90ce217c63" exitCode=0
Oct 31 22:53:43.341042 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:43.340654    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q" event={"ID":"19ea0f64-10fa-438a-aa25-724be70d152f","Type":"ContainerDied","Data":"fe0ddd63abe231ef665ac0a12652a39fa4c8dae5feaa942dee60bd90ce217c63"}
Oct 31 22:53:44.345746 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:44.345707    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q" event={"ID":"19ea0f64-10fa-438a-aa25-724be70d152f","Type":"ContainerStarted","Data":"1223cff5f967ef4870d2d05d3cf3da1a15426640988da02b51223b40dbe47665"}
Oct 31 22:53:44.346126 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:44.345850    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q"
Oct 31 22:53:44.387104 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:44.387055    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q" podStartSLOduration=2.792347393 podStartE2EDuration="6.387040657s" totalImagesPullingTime="3.154240935s" totalInitContainerRuntime="440.452329ms" isStatefulPod=false podCreationTimestamp="2026-10-31 22:53:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:53:44.385416135 +0000 UTC m=+243.132605641" watchObservedRunningTime="2026-10-31 22:53:44.387040657 +0000 UTC m=+243.134230161"
Oct 31 22:53:55.352134 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:53:55.352101    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q"
Oct 31 22:54:02.544222 ip-10-0-135-163 kubenswrapper[4301]: E1031 22:54:02.544180    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 22:54:15.419621 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:54:15.419590    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-689d5458bb-hzp7b"]
Oct 31 22:54:15.424771 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:54:15.424748    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-689d5458bb-hzp7b"
Oct 31 22:54:15.431996 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:54:15.431980    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 31 22:54:15.432073 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:54:15.431984    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 31 22:54:15.435926 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:54:15.435911    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7xzl9\""
Oct 31 22:54:15.437637 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:54:15.437617    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-689d5458bb-hzp7b"]
Oct 31 22:54:15.503166 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:54:15.503137    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hczzk\" (UniqueName: \"kubernetes.io/projected/7230a094-2ba4-41af-8bec-2d8b2b266ca9-kube-api-access-hczzk\") pod \"kube-auth-proxy-689d5458bb-hzp7b\" (UID: \"7230a094-2ba4-41af-8bec-2d8b2b266ca9\") " pod="openshift-ingress/kube-auth-proxy-689d5458bb-hzp7b"
Oct 31 22:54:15.503326 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:54:15.503189    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7230a094-2ba4-41af-8bec-2d8b2b266ca9-tls-certs\") pod \"kube-auth-proxy-689d5458bb-hzp7b\" (UID: \"7230a094-2ba4-41af-8bec-2d8b2b266ca9\") " pod="openshift-ingress/kube-auth-proxy-689d5458bb-hzp7b"
Oct 31 22:54:15.503326 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:54:15.503209    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7230a094-2ba4-41af-8bec-2d8b2b266ca9-tmp\") pod \"kube-auth-proxy-689d5458bb-hzp7b\" (UID: \"7230a094-2ba4-41af-8bec-2d8b2b266ca9\") " pod="openshift-ingress/kube-auth-proxy-689d5458bb-hzp7b"
Oct 31 22:54:15.604105 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:54:15.604074    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7230a094-2ba4-41af-8bec-2d8b2b266ca9-tls-certs\") pod \"kube-auth-proxy-689d5458bb-hzp7b\" (UID: \"7230a094-2ba4-41af-8bec-2d8b2b266ca9\") " pod="openshift-ingress/kube-auth-proxy-689d5458bb-hzp7b"
Oct 31 22:54:15.604105 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:54:15.604105    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7230a094-2ba4-41af-8bec-2d8b2b266ca9-tmp\") pod \"kube-auth-proxy-689d5458bb-hzp7b\" (UID: \"7230a094-2ba4-41af-8bec-2d8b2b266ca9\") " pod="openshift-ingress/kube-auth-proxy-689d5458bb-hzp7b"
Oct 31 22:54:15.604301 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:54:15.604157    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hczzk\" (UniqueName: \"kubernetes.io/projected/7230a094-2ba4-41af-8bec-2d8b2b266ca9-kube-api-access-hczzk\") pod \"kube-auth-proxy-689d5458bb-hzp7b\" (UID: \"7230a094-2ba4-41af-8bec-2d8b2b266ca9\") " pod="openshift-ingress/kube-auth-proxy-689d5458bb-hzp7b"
Oct 31 22:54:15.606521 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:54:15.606494    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7230a094-2ba4-41af-8bec-2d8b2b266ca9-tmp\") pod \"kube-auth-proxy-689d5458bb-hzp7b\" (UID: \"7230a094-2ba4-41af-8bec-2d8b2b266ca9\") " pod="openshift-ingress/kube-auth-proxy-689d5458bb-hzp7b"
Oct 31 22:54:15.606656 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:54:15.606583    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7230a094-2ba4-41af-8bec-2d8b2b266ca9-tls-certs\") pod \"kube-auth-proxy-689d5458bb-hzp7b\" (UID: \"7230a094-2ba4-41af-8bec-2d8b2b266ca9\") " pod="openshift-ingress/kube-auth-proxy-689d5458bb-hzp7b"
Oct 31 22:54:15.622998 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:54:15.622972    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hczzk\" (UniqueName: \"kubernetes.io/projected/7230a094-2ba4-41af-8bec-2d8b2b266ca9-kube-api-access-hczzk\") pod \"kube-auth-proxy-689d5458bb-hzp7b\" (UID: \"7230a094-2ba4-41af-8bec-2d8b2b266ca9\") " pod="openshift-ingress/kube-auth-proxy-689d5458bb-hzp7b"
Oct 31 22:54:15.735086 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:54:15.735012    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-689d5458bb-hzp7b"
Oct 31 22:54:16.008150 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:54:16.008064    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-689d5458bb-hzp7b"]
Oct 31 22:54:16.011662 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:54:16.011627    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7230a094_2ba4_41af_8bec_2d8b2b266ca9.slice/crio-3c7a668db5fd48be8aba0827d8f4d87d0bea431eed7d340e6cce92699ddebd6c WatchSource:0}: Error finding container 3c7a668db5fd48be8aba0827d8f4d87d0bea431eed7d340e6cce92699ddebd6c: Status 404 returned error can't find the container with id 3c7a668db5fd48be8aba0827d8f4d87d0bea431eed7d340e6cce92699ddebd6c
Oct 31 22:54:16.458028 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:54:16.457996    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-689d5458bb-hzp7b" event={"ID":"7230a094-2ba4-41af-8bec-2d8b2b266ca9","Type":"ContainerStarted","Data":"3c7a668db5fd48be8aba0827d8f4d87d0bea431eed7d340e6cce92699ddebd6c"}
Oct 31 22:54:18.468055 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:54:18.468022    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-689d5458bb-hzp7b" event={"ID":"7230a094-2ba4-41af-8bec-2d8b2b266ca9","Type":"ContainerStarted","Data":"fd4daaf429e36e8dddeead7253d23874042e9056544bf2aa2dd20c4d2699f53d"}
Oct 31 22:54:18.487417 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:54:18.487349    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-689d5458bb-hzp7b" podStartSLOduration=1.142058099 podStartE2EDuration="3.487337315s" totalImagesPullingTime="2.345279216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:54:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:54:18.486501274 +0000 UTC m=+277.233690778" watchObservedRunningTime="2026-10-31 22:54:18.487337315 +0000 UTC m=+277.234526818"
Oct 31 22:54:32.544649 ip-10-0-135-163 kubenswrapper[4301]: E1031 22:54:32.544610    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 22:54:41.403393 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:54:41.403346    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s9nl_1f0042ee-6258-4aa8-af48-7568f2f2e8a1/ovn-acl-logging/0.log"
Oct 31 22:54:41.403824 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:54:41.403618    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s9nl_1f0042ee-6258-4aa8-af48-7568f2f2e8a1/ovn-acl-logging/0.log"
Oct 31 22:55:02.544331 ip-10-0-135-163 kubenswrapper[4301]: E1031 22:55:02.544243    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 22:55:10.370716 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.370677    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-pmrkc"]
Oct 31 22:55:10.373888 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.373870    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-pmrkc"
Oct 31 22:55:10.387442 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.387414    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-pmrkc"]
Oct 31 22:55:10.397234 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.397208    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-srxt8\""
Oct 31 22:55:10.397391 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.397257    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 31 22:55:10.404788 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.404767    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 31 22:55:10.404928 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.404917    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 31 22:55:10.421822 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.421798    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5bjx\" (UniqueName: \"kubernetes.io/projected/edcb76cb-2a03-451b-a920-209041f60ded-kube-api-access-t5bjx\") pod \"llmisvc-controller-manager-869d5dcc5d-pmrkc\" (UID: \"edcb76cb-2a03-451b-a920-209041f60ded\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-pmrkc"
Oct 31 22:55:10.421945 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.421840    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edcb76cb-2a03-451b-a920-209041f60ded-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-pmrkc\" (UID: \"edcb76cb-2a03-451b-a920-209041f60ded\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-pmrkc"
Oct 31 22:55:10.424302 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.424275    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-nrdhv"]
Oct 31 22:55:10.428674 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.428657    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-nrdhv"
Oct 31 22:55:10.438893 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.438872    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-nrdhv"]
Oct 31 22:55:10.456667 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.456645    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-6tl4h\""
Oct 31 22:55:10.456803 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.456722    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 31 22:55:10.456803 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.456791    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 31 22:55:10.522842 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.522811    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5bjx\" (UniqueName: \"kubernetes.io/projected/edcb76cb-2a03-451b-a920-209041f60ded-kube-api-access-t5bjx\") pod \"llmisvc-controller-manager-869d5dcc5d-pmrkc\" (UID: \"edcb76cb-2a03-451b-a920-209041f60ded\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-pmrkc"
Oct 31 22:55:10.523014 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.522851    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2e1fa43-f1e5-4da5-a89e-cb3ab82eb88e-cert\") pod \"odh-model-controller-769875db56-nrdhv\" (UID: \"d2e1fa43-f1e5-4da5-a89e-cb3ab82eb88e\") " pod="opendatahub/odh-model-controller-769875db56-nrdhv"
Oct 31 22:55:10.523014 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.522886    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2e1fa43-f1e5-4da5-a89e-cb3ab82eb88e-metrics-certs\") pod \"odh-model-controller-769875db56-nrdhv\" (UID: \"d2e1fa43-f1e5-4da5-a89e-cb3ab82eb88e\") " pod="opendatahub/odh-model-controller-769875db56-nrdhv"
Oct 31 22:55:10.523014 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.522909    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttx4f\" (UniqueName: \"kubernetes.io/projected/d2e1fa43-f1e5-4da5-a89e-cb3ab82eb88e-kube-api-access-ttx4f\") pod \"odh-model-controller-769875db56-nrdhv\" (UID: \"d2e1fa43-f1e5-4da5-a89e-cb3ab82eb88e\") " pod="opendatahub/odh-model-controller-769875db56-nrdhv"
Oct 31 22:55:10.523014 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.522939    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edcb76cb-2a03-451b-a920-209041f60ded-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-pmrkc\" (UID: \"edcb76cb-2a03-451b-a920-209041f60ded\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-pmrkc"
Oct 31 22:55:10.525347 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.525320    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edcb76cb-2a03-451b-a920-209041f60ded-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-pmrkc\" (UID: \"edcb76cb-2a03-451b-a920-209041f60ded\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-pmrkc"
Oct 31 22:55:10.534961 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.534939    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5bjx\" (UniqueName: \"kubernetes.io/projected/edcb76cb-2a03-451b-a920-209041f60ded-kube-api-access-t5bjx\") pod \"llmisvc-controller-manager-869d5dcc5d-pmrkc\" (UID: \"edcb76cb-2a03-451b-a920-209041f60ded\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-pmrkc"
Oct 31 22:55:10.613423 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.613390    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g"]
Oct 31 22:55:10.616553 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.616538    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g"
Oct 31 22:55:10.623714 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.623692    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2e1fa43-f1e5-4da5-a89e-cb3ab82eb88e-cert\") pod \"odh-model-controller-769875db56-nrdhv\" (UID: \"d2e1fa43-f1e5-4da5-a89e-cb3ab82eb88e\") " pod="opendatahub/odh-model-controller-769875db56-nrdhv"
Oct 31 22:55:10.623821 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.623734    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2e1fa43-f1e5-4da5-a89e-cb3ab82eb88e-metrics-certs\") pod \"odh-model-controller-769875db56-nrdhv\" (UID: \"d2e1fa43-f1e5-4da5-a89e-cb3ab82eb88e\") " pod="opendatahub/odh-model-controller-769875db56-nrdhv"
Oct 31 22:55:10.623885 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.623832    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttx4f\" (UniqueName: \"kubernetes.io/projected/d2e1fa43-f1e5-4da5-a89e-cb3ab82eb88e-kube-api-access-ttx4f\") pod \"odh-model-controller-769875db56-nrdhv\" (UID: \"d2e1fa43-f1e5-4da5-a89e-cb3ab82eb88e\") " pod="opendatahub/odh-model-controller-769875db56-nrdhv"
Oct 31 22:55:10.626141 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.626119    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2e1fa43-f1e5-4da5-a89e-cb3ab82eb88e-cert\") pod \"odh-model-controller-769875db56-nrdhv\" (UID: \"d2e1fa43-f1e5-4da5-a89e-cb3ab82eb88e\") " pod="opendatahub/odh-model-controller-769875db56-nrdhv"
Oct 31 22:55:10.626398 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.626378    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2e1fa43-f1e5-4da5-a89e-cb3ab82eb88e-metrics-certs\") pod \"odh-model-controller-769875db56-nrdhv\" (UID: \"d2e1fa43-f1e5-4da5-a89e-cb3ab82eb88e\") " pod="opendatahub/odh-model-controller-769875db56-nrdhv"
Oct 31 22:55:10.643725 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.643699    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g"]
Oct 31 22:55:10.651705 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.651682    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttx4f\" (UniqueName: \"kubernetes.io/projected/d2e1fa43-f1e5-4da5-a89e-cb3ab82eb88e-kube-api-access-ttx4f\") pod \"odh-model-controller-769875db56-nrdhv\" (UID: \"d2e1fa43-f1e5-4da5-a89e-cb3ab82eb88e\") " pod="opendatahub/odh-model-controller-769875db56-nrdhv"
Oct 31 22:55:10.687485 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.687459    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-pmrkc"
Oct 31 22:55:10.732448 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.732401    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/396eb171-d982-4c65-8ce4-0726cb0e29e5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dd96g\" (UID: \"396eb171-d982-4c65-8ce4-0726cb0e29e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g"
Oct 31 22:55:10.732978 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.732950    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/396eb171-d982-4c65-8ce4-0726cb0e29e5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dd96g\" (UID: \"396eb171-d982-4c65-8ce4-0726cb0e29e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g"
Oct 31 22:55:10.733101 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.733029    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/396eb171-d982-4c65-8ce4-0726cb0e29e5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dd96g\" (UID: \"396eb171-d982-4c65-8ce4-0726cb0e29e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g"
Oct 31 22:55:10.733101 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.733068    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/396eb171-d982-4c65-8ce4-0726cb0e29e5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dd96g\" (UID: \"396eb171-d982-4c65-8ce4-0726cb0e29e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g"
Oct 31 22:55:10.733101 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.733089    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/396eb171-d982-4c65-8ce4-0726cb0e29e5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dd96g\" (UID: \"396eb171-d982-4c65-8ce4-0726cb0e29e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g"
Oct 31 22:55:10.733665 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.733104    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrrqq\" (UniqueName: \"kubernetes.io/projected/396eb171-d982-4c65-8ce4-0726cb0e29e5-kube-api-access-jrrqq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dd96g\" (UID: \"396eb171-d982-4c65-8ce4-0726cb0e29e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g"
Oct 31 22:55:10.733665 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.733124    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/396eb171-d982-4c65-8ce4-0726cb0e29e5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dd96g\" (UID: \"396eb171-d982-4c65-8ce4-0726cb0e29e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g"
Oct 31 22:55:10.741719 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.741689    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-nrdhv"
Oct 31 22:55:10.833888 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.833855    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/396eb171-d982-4c65-8ce4-0726cb0e29e5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dd96g\" (UID: \"396eb171-d982-4c65-8ce4-0726cb0e29e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g"
Oct 31 22:55:10.834028 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.833893    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/396eb171-d982-4c65-8ce4-0726cb0e29e5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dd96g\" (UID: \"396eb171-d982-4c65-8ce4-0726cb0e29e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g"
Oct 31 22:55:10.834028 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.833912    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrrqq\" (UniqueName: \"kubernetes.io/projected/396eb171-d982-4c65-8ce4-0726cb0e29e5-kube-api-access-jrrqq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dd96g\" (UID: \"396eb171-d982-4c65-8ce4-0726cb0e29e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g"
Oct 31 22:55:10.834028 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.833936    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/396eb171-d982-4c65-8ce4-0726cb0e29e5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dd96g\" (UID: \"396eb171-d982-4c65-8ce4-0726cb0e29e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g"
Oct 31 22:55:10.834028 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.833987    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/396eb171-d982-4c65-8ce4-0726cb0e29e5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dd96g\" (UID: \"396eb171-d982-4c65-8ce4-0726cb0e29e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g"
Oct 31 22:55:10.834028 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.834011    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/396eb171-d982-4c65-8ce4-0726cb0e29e5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dd96g\" (UID: \"396eb171-d982-4c65-8ce4-0726cb0e29e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g"
Oct 31 22:55:10.834194 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.834083    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/396eb171-d982-4c65-8ce4-0726cb0e29e5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dd96g\" (UID: \"396eb171-d982-4c65-8ce4-0726cb0e29e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g"
Oct 31 22:55:10.834816 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.834793    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/396eb171-d982-4c65-8ce4-0726cb0e29e5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dd96g\" (UID: \"396eb171-d982-4c65-8ce4-0726cb0e29e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g"
Oct 31 22:55:10.836255 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.836230    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/396eb171-d982-4c65-8ce4-0726cb0e29e5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dd96g\" (UID: \"396eb171-d982-4c65-8ce4-0726cb0e29e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g"
Oct 31 22:55:10.836400 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.836260    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/396eb171-d982-4c65-8ce4-0726cb0e29e5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dd96g\" (UID: \"396eb171-d982-4c65-8ce4-0726cb0e29e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g"
Oct 31 22:55:10.836541 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.836525    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/396eb171-d982-4c65-8ce4-0726cb0e29e5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dd96g\" (UID: \"396eb171-d982-4c65-8ce4-0726cb0e29e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g"
Oct 31 22:55:10.836644 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.836628    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/396eb171-d982-4c65-8ce4-0726cb0e29e5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dd96g\" (UID: \"396eb171-d982-4c65-8ce4-0726cb0e29e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g"
Oct 31 22:55:10.846806 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.846781    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/396eb171-d982-4c65-8ce4-0726cb0e29e5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dd96g\" (UID: \"396eb171-d982-4c65-8ce4-0726cb0e29e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g"
Oct 31 22:55:10.847058 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.847042    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrrqq\" (UniqueName: \"kubernetes.io/projected/396eb171-d982-4c65-8ce4-0726cb0e29e5-kube-api-access-jrrqq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dd96g\" (UID: \"396eb171-d982-4c65-8ce4-0726cb0e29e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g"
Oct 31 22:55:10.928244 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.928169    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g"
Oct 31 22:55:10.959474 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.959444    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-pmrkc"]
Oct 31 22:55:10.962748 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:55:10.962723    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podedcb76cb_2a03_451b_a920_209041f60ded.slice/crio-67a0c8027e60b704e7e9024e945f5fd00ea78dd4a0e67562529829348b490fae WatchSource:0}: Error finding container 67a0c8027e60b704e7e9024e945f5fd00ea78dd4a0e67562529829348b490fae: Status 404 returned error can't find the container with id 67a0c8027e60b704e7e9024e945f5fd00ea78dd4a0e67562529829348b490fae
Oct 31 22:55:10.964010 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:10.963990    4301 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 22:55:11.017422 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:11.017392    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-nrdhv"]
Oct 31 22:55:11.020494 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:55:11.020471    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e1fa43_f1e5_4da5_a89e_cb3ab82eb88e.slice/crio-da8c9a3a384789ea18647b1338c86835f6e17e372fabdea43200774ded538707 WatchSource:0}: Error finding container da8c9a3a384789ea18647b1338c86835f6e17e372fabdea43200774ded538707: Status 404 returned error can't find the container with id da8c9a3a384789ea18647b1338c86835f6e17e372fabdea43200774ded538707
Oct 31 22:55:11.206065 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:11.205997    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g"]
Oct 31 22:55:11.208794 ip-10-0-135-163 kubenswrapper[4301]: W1031 22:55:11.208759    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod396eb171_d982_4c65_8ce4_0726cb0e29e5.slice/crio-72ca877f0c298e67aed2a4a1fa44f780f0133f285ede175c02b58daff7f84ab8 WatchSource:0}: Error finding container 72ca877f0c298e67aed2a4a1fa44f780f0133f285ede175c02b58daff7f84ab8: Status 404 returned error can't find the container with id 72ca877f0c298e67aed2a4a1fa44f780f0133f285ede175c02b58daff7f84ab8
Oct 31 22:55:11.213530 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:11.213458    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 22:55:11.213646 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:11.213572    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 22:55:11.671824 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:11.671785    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-nrdhv" event={"ID":"d2e1fa43-f1e5-4da5-a89e-cb3ab82eb88e","Type":"ContainerStarted","Data":"da8c9a3a384789ea18647b1338c86835f6e17e372fabdea43200774ded538707"}
Oct 31 22:55:11.674677 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:11.674648    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g" event={"ID":"396eb171-d982-4c65-8ce4-0726cb0e29e5","Type":"ContainerStarted","Data":"c195a186a6db22b8b8955fd82bfb2d12f616a68c07003ccf851a19073df4bfa6"}
Oct 31 22:55:11.674805 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:11.674745    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g" event={"ID":"396eb171-d982-4c65-8ce4-0726cb0e29e5","Type":"ContainerStarted","Data":"72ca877f0c298e67aed2a4a1fa44f780f0133f285ede175c02b58daff7f84ab8"}
Oct 31 22:55:11.674805 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:11.674771    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g"
Oct 31 22:55:11.676701 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:11.676641    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-pmrkc" event={"ID":"edcb76cb-2a03-451b-a920-209041f60ded","Type":"ContainerStarted","Data":"67a0c8027e60b704e7e9024e945f5fd00ea78dd4a0e67562529829348b490fae"}
Oct 31 22:55:11.699845 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:11.699794    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g" podStartSLOduration=1.699778493 podStartE2EDuration="1.699778493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:55:11.699494236 +0000 UTC m=+330.446683742" watchObservedRunningTime="2026-10-31 22:55:11.699778493 +0000 UTC m=+330.446967997"
Oct 31 22:55:12.684897 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:12.684867    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g"
Oct 31 22:55:12.752632 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:12.752596    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565"]
Oct 31 22:55:12.752889 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:12.752846    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565" podUID="f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2" containerName="discovery" containerID="cri-o://d8bed165fc00064230a86de05b6d9fd2dba5edb6810255792a2b35793a441ce2" gracePeriod=30
Oct 31 22:55:13.034388 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.034340    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565"
Oct 31 22:55:13.153420 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.153393    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-istio-csr-dns-cert\") pod \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\" (UID: \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\") "
Oct 31 22:55:13.153580 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.153437    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-istio-token\" (UniqueName: \"kubernetes.io/projected/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-istio-token\") pod \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\" (UID: \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\") "
Oct 31 22:55:13.153580 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.153471    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-istio-kubeconfig\") pod \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\" (UID: \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\") "
Oct 31 22:55:13.153901 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.153868    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-kube-api-access-mlhq7\" (UniqueName: \"kubernetes.io/projected/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-kube-api-access-mlhq7\") pod \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\" (UID: \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\") "
Oct 31 22:55:13.154016 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.153942    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-istio-csr-ca-configmap\") pod \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\" (UID: \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\") "
Oct 31 22:55:13.154016 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.153980    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-local-certs\") pod \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\" (UID: \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\") "
Oct 31 22:55:13.154128 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.154049    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-cacerts\" (UniqueName: \"kubernetes.io/secret/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-cacerts\") pod \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\" (UID: \"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2\") "
Oct 31 22:55:13.154334 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.154311    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-istio-csr-ca-configmap" pod "f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2" (UID: "f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 22:55:13.156374 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.156319    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-istio-csr-dns-cert" pod "f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2" (UID: "f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 22:55:13.156485 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.156376    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-istio-token" pod "f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2" (UID: "f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 22:55:13.156485 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.156445    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-istio-kubeconfig" pod "f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2" (UID: "f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 22:55:13.156746 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.156722    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-cacerts" pod "f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2" (UID: "f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 22:55:13.156830 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.156785    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-kube-api-access-mlhq7" pod "f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2" (UID: "f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2"). InnerVolumeSpecName "kube-api-access-mlhq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 22:55:13.156830 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.156786    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-local-certs" pod "f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2" (UID: "f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 22:55:13.255239 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.255210    4301 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-cacerts\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 22:55:13.255388 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.255244    4301 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-istio-csr-dns-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 22:55:13.255388 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.255256    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 22:55:13.255388 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.255265    4301 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-istio-kubeconfig\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 22:55:13.255388 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.255274    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlhq7\" (UniqueName: \"kubernetes.io/projected/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-kube-api-access-mlhq7\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 22:55:13.255388 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.255287    4301 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-istio-csr-ca-configmap\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 22:55:13.255388 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.255297    4301 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2-local-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 22:55:13.689292 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.689232    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-pmrkc" event={"ID":"edcb76cb-2a03-451b-a920-209041f60ded","Type":"ContainerStarted","Data":"bfc8c92e9786720c901370a475a5c899a68e01b941cbad95e84acd807b766ee7"}
Oct 31 22:55:13.689780 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.689314    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-pmrkc"
Oct 31 22:55:13.690740 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.690707    4301 generic.go:356] "Generic (PLEG): container finished" podID="f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2" containerID="d8bed165fc00064230a86de05b6d9fd2dba5edb6810255792a2b35793a441ce2" exitCode=0
Oct 31 22:55:13.690865 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.690748    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565"
Oct 31 22:55:13.690865 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.690776    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565" event={"ID":"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2","Type":"ContainerDied","Data":"d8bed165fc00064230a86de05b6d9fd2dba5edb6810255792a2b35793a441ce2"}
Oct 31 22:55:13.690865 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.690815    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565" event={"ID":"f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2","Type":"ContainerDied","Data":"b7c2ec876a66ecf8bf76bb24a1486434ce54d581b58c59fd5229c89265c03d08"}
Oct 31 22:55:13.690865 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.690841    4301 scope.go:122] "RemoveContainer" containerID="d8bed165fc00064230a86de05b6d9fd2dba5edb6810255792a2b35793a441ce2"
Oct 31 22:55:13.700614 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.700597    4301 scope.go:122] "RemoveContainer" containerID="d8bed165fc00064230a86de05b6d9fd2dba5edb6810255792a2b35793a441ce2"
Oct 31 22:55:13.700902 ip-10-0-135-163 kubenswrapper[4301]: E1031 22:55:13.700877    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8bed165fc00064230a86de05b6d9fd2dba5edb6810255792a2b35793a441ce2\": container with ID starting with d8bed165fc00064230a86de05b6d9fd2dba5edb6810255792a2b35793a441ce2 not found: ID does not exist" containerID="d8bed165fc00064230a86de05b6d9fd2dba5edb6810255792a2b35793a441ce2"
Oct 31 22:55:13.700966 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.700911    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8bed165fc00064230a86de05b6d9fd2dba5edb6810255792a2b35793a441ce2"} err="failed to get container status \"d8bed165fc00064230a86de05b6d9fd2dba5edb6810255792a2b35793a441ce2\": rpc error: code = NotFound desc = could not find container \"d8bed165fc00064230a86de05b6d9fd2dba5edb6810255792a2b35793a441ce2\": container with ID starting with d8bed165fc00064230a86de05b6d9fd2dba5edb6810255792a2b35793a441ce2 not found: ID does not exist"
Oct 31 22:55:13.708015 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.707964    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-pmrkc" podStartSLOduration=1.573450244 podStartE2EDuration="3.707950847s" totalImagesPullingTime="2.134500603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:55:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:55:13.70669575 +0000 UTC m=+332.453885449" watchObservedRunningTime="2026-10-31 22:55:13.707950847 +0000 UTC m=+332.455140353"
Oct 31 22:55:13.719825 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.719801    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565"]
Oct 31 22:55:13.725199 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:13.725176    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-k6565"]
Oct 31 22:55:14.696878 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:14.696843    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-nrdhv" event={"ID":"d2e1fa43-f1e5-4da5-a89e-cb3ab82eb88e","Type":"ContainerStarted","Data":"2286da34953cab8335e11bded058f6a8dbdb2b295a1b0cc661f8258ecff98901"}
Oct 31 22:55:14.716221 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:14.716172    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-nrdhv" podStartSLOduration=1.96996511 podStartE2EDuration="4.716158272s" totalImagesPullingTime="2.746193162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:55:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:55:14.714895027 +0000 UTC m=+333.462084522" watchObservedRunningTime="2026-10-31 22:55:14.716158272 +0000 UTC m=+333.463347776"
Oct 31 22:55:15.402324 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:15.402288    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2" path="/var/lib/kubelet/pods/f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2/volumes"
Oct 31 22:55:15.701141 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:15.701066    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-nrdhv"
Oct 31 22:55:26.707771 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:26.707742    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-nrdhv"
Oct 31 22:55:32.544774 ip-10-0-135-163 kubenswrapper[4301]: E1031 22:55:32.544737    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 22:55:44.698927 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:55:44.698888    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-pmrkc"
Oct 31 22:56:02.544428 ip-10-0-135-163 kubenswrapper[4301]: E1031 22:56:02.544391    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 22:56:32.544305 ip-10-0-135-163 kubenswrapper[4301]: E1031 22:56:32.544215    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 22:57:02.544094 ip-10-0-135-163 kubenswrapper[4301]: E1031 22:57:02.544061    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 22:57:32.544109 ip-10-0-135-163 kubenswrapper[4301]: E1031 22:57:32.544072    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 22:58:02.544426 ip-10-0-135-163 kubenswrapper[4301]: E1031 22:58:02.544333    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 22:58:32.544279 ip-10-0-135-163 kubenswrapper[4301]: E1031 22:58:32.544236    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 22:59:02.545055 ip-10-0-135-163 kubenswrapper[4301]: E1031 22:59:02.544951    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 22:59:32.545057 ip-10-0-135-163 kubenswrapper[4301]: E1031 22:59:32.544983    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 22:59:41.430017 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:59:41.429988    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s9nl_1f0042ee-6258-4aa8-af48-7568f2f2e8a1/ovn-acl-logging/0.log"
Oct 31 22:59:41.430497 ip-10-0-135-163 kubenswrapper[4301]: I1031 22:59:41.430069    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s9nl_1f0042ee-6258-4aa8-af48-7568f2f2e8a1/ovn-acl-logging/0.log"
Oct 31 23:00:02.544879 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:00:02.544838    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:00:32.544175 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:00:32.544133    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:01:02.544384 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:01:02.544265    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:01:32.544394 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:01:32.544339    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:02:02.544992 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:02:02.544954    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:02:32.544478 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:02:32.544401    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:03:02.544838 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:03:02.544802    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:03:32.543590 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:03:32.543548    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:04:02.544200 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:04:02.544165    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:04:32.545300 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:04:32.545261    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:04:41.452652 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:41.452626    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s9nl_1f0042ee-6258-4aa8-af48-7568f2f2e8a1/ovn-acl-logging/0.log"
Oct 31 23:04:41.453474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:41.453457    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s9nl_1f0042ee-6258-4aa8-af48-7568f2f2e8a1/ovn-acl-logging/0.log"
Oct 31 23:04:53.361450 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.361414    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-lh6zd"]
Oct 31 23:04:53.362027 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.361844    4301 cpu_manager.go:414] "containerMap: removing container" podUID="f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2" containerName="discovery"
Oct 31 23:04:53.362027 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.361863    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2" containerName="discovery"
Oct 31 23:04:53.362027 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.361935    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="f733ecfd-bd0b-41d4-b4b1-d6b929bfb6c2" containerName="discovery"
Oct 31 23:04:53.364881 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.364860    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lh6zd"
Oct 31 23:04:53.375750 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.375722    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-lh6zd"]
Oct 31 23:04:53.388475 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.388439    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kjfr7\""
Oct 31 23:04:53.388475 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.388439    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 31 23:04:53.388475 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.388435    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 31 23:04:53.388669 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.388450    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 23:04:53.388669 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.388464    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 23:04:53.498465 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.498431    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/85434a39-a2bd-43e2-980e-2983c549f656-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-lh6zd\" (UID: \"85434a39-a2bd-43e2-980e-2983c549f656\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lh6zd"
Oct 31 23:04:53.498658 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.498492    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/85434a39-a2bd-43e2-980e-2983c549f656-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-lh6zd\" (UID: \"85434a39-a2bd-43e2-980e-2983c549f656\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lh6zd"
Oct 31 23:04:53.498658 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.498579    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvcw7\" (UniqueName: \"kubernetes.io/projected/85434a39-a2bd-43e2-980e-2983c549f656-kube-api-access-jvcw7\") pod \"kuadrant-console-plugin-66c4fb575d-lh6zd\" (UID: \"85434a39-a2bd-43e2-980e-2983c549f656\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lh6zd"
Oct 31 23:04:53.599385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.599330    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/85434a39-a2bd-43e2-980e-2983c549f656-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-lh6zd\" (UID: \"85434a39-a2bd-43e2-980e-2983c549f656\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lh6zd"
Oct 31 23:04:53.599581 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.599399    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvcw7\" (UniqueName: \"kubernetes.io/projected/85434a39-a2bd-43e2-980e-2983c549f656-kube-api-access-jvcw7\") pod \"kuadrant-console-plugin-66c4fb575d-lh6zd\" (UID: \"85434a39-a2bd-43e2-980e-2983c549f656\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lh6zd"
Oct 31 23:04:53.599581 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.599468    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/85434a39-a2bd-43e2-980e-2983c549f656-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-lh6zd\" (UID: \"85434a39-a2bd-43e2-980e-2983c549f656\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lh6zd"
Oct 31 23:04:53.600120 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.600095    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/85434a39-a2bd-43e2-980e-2983c549f656-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-lh6zd\" (UID: \"85434a39-a2bd-43e2-980e-2983c549f656\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lh6zd"
Oct 31 23:04:53.601844 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.601819    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:04:53.602304 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.602285    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/85434a39-a2bd-43e2-980e-2983c549f656-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-lh6zd\" (UID: \"85434a39-a2bd-43e2-980e-2983c549f656\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lh6zd"
Oct 31 23:04:53.605047 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.605031    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-64lft"
Oct 31 23:04:53.608041 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.608024    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 31 23:04:53.612913 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.612838    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvcw7\" (UniqueName: \"kubernetes.io/projected/85434a39-a2bd-43e2-980e-2983c549f656-kube-api-access-jvcw7\") pod \"kuadrant-console-plugin-66c4fb575d-lh6zd\" (UID: \"85434a39-a2bd-43e2-980e-2983c549f656\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lh6zd"
Oct 31 23:04:53.614903 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.614883    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:04:53.648658 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.648628    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:04:53.689508 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.689480    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lh6zd"
Oct 31 23:04:53.700759 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.700732    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r5tw\" (UniqueName: \"kubernetes.io/projected/800e23f4-edde-455c-bd92-acb8ad2cc8af-kube-api-access-7r5tw\") pod \"limitador-limitador-7c9b87d574-64lft\" (UID: \"800e23f4-edde-455c-bd92-acb8ad2cc8af\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-64lft"
Oct 31 23:04:53.700826 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.700779    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/800e23f4-edde-455c-bd92-acb8ad2cc8af-config-file\") pod \"limitador-limitador-7c9b87d574-64lft\" (UID: \"800e23f4-edde-455c-bd92-acb8ad2cc8af\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-64lft"
Oct 31 23:04:53.801604 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.801560    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r5tw\" (UniqueName: \"kubernetes.io/projected/800e23f4-edde-455c-bd92-acb8ad2cc8af-kube-api-access-7r5tw\") pod \"limitador-limitador-7c9b87d574-64lft\" (UID: \"800e23f4-edde-455c-bd92-acb8ad2cc8af\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-64lft"
Oct 31 23:04:53.801763 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.801614    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/800e23f4-edde-455c-bd92-acb8ad2cc8af-config-file\") pod \"limitador-limitador-7c9b87d574-64lft\" (UID: \"800e23f4-edde-455c-bd92-acb8ad2cc8af\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-64lft"
Oct 31 23:04:53.802269 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.802250    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/800e23f4-edde-455c-bd92-acb8ad2cc8af-config-file\") pod \"limitador-limitador-7c9b87d574-64lft\" (UID: \"800e23f4-edde-455c-bd92-acb8ad2cc8af\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-64lft"
Oct 31 23:04:53.812849 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.812828    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r5tw\" (UniqueName: \"kubernetes.io/projected/800e23f4-edde-455c-bd92-acb8ad2cc8af-kube-api-access-7r5tw\") pod \"limitador-limitador-7c9b87d574-64lft\" (UID: \"800e23f4-edde-455c-bd92-acb8ad2cc8af\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-64lft"
Oct 31 23:04:53.927652 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.927628    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-64lft"
Oct 31 23:04:53.961281 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.961248    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-lh6zd"]
Oct 31 23:04:53.964330 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:04:53.964299    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85434a39_a2bd_43e2_980e_2983c549f656.slice/crio-83cda040d492ab3500fe837388ad329beed3a6762e2380724a4bafd102d9c8fc WatchSource:0}: Error finding container 83cda040d492ab3500fe837388ad329beed3a6762e2380724a4bafd102d9c8fc: Status 404 returned error can't find the container with id 83cda040d492ab3500fe837388ad329beed3a6762e2380724a4bafd102d9c8fc
Oct 31 23:04:53.965845 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:53.965823    4301 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 23:04:54.201830 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:54.201755    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:04:54.205012 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:04:54.204990    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod800e23f4_edde_455c_bd92_acb8ad2cc8af.slice/crio-ea4b6289771823d7f0f72627838bb36e66d5857c489c41f46286585d007e622f WatchSource:0}: Error finding container ea4b6289771823d7f0f72627838bb36e66d5857c489c41f46286585d007e622f: Status 404 returned error can't find the container with id ea4b6289771823d7f0f72627838bb36e66d5857c489c41f46286585d007e622f
Oct 31 23:04:54.902181 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:54.902145    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-64lft" event={"ID":"800e23f4-edde-455c-bd92-acb8ad2cc8af","Type":"ContainerStarted","Data":"ea4b6289771823d7f0f72627838bb36e66d5857c489c41f46286585d007e622f"}
Oct 31 23:04:54.903221 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:54.903196    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lh6zd" event={"ID":"85434a39-a2bd-43e2-980e-2983c549f656","Type":"ContainerStarted","Data":"83cda040d492ab3500fe837388ad329beed3a6762e2380724a4bafd102d9c8fc"}
Oct 31 23:04:57.921756 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:57.921723    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-64lft" event={"ID":"800e23f4-edde-455c-bd92-acb8ad2cc8af","Type":"ContainerStarted","Data":"d3e0a5174bf056e9ab5eef87bfcbcf01f3efdbc34d543e21da61b83da6079efa"}
Oct 31 23:04:57.922203 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:57.921850    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-64lft"
Oct 31 23:04:57.938275 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:57.938125    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-64lft" podStartSLOduration=2.066919471 podStartE2EDuration="4.938108026s" totalImagesPullingTime="2.871188555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:04:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:04:57.937714637 +0000 UTC m=+916.684904154" watchObservedRunningTime="2026-10-31 23:04:57.938108026 +0000 UTC m=+916.685297531"
Oct 31 23:04:59.931836 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:59.931801    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lh6zd" event={"ID":"85434a39-a2bd-43e2-980e-2983c549f656","Type":"ContainerStarted","Data":"7d7b8c09de31fb25cbcd242981f187db00dd9afc6fd1bd497d3fdce19c926797"}
Oct 31 23:04:59.949090 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:04:59.949038    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lh6zd" podStartSLOduration=1.676870082 podStartE2EDuration="6.949024226s" totalImagesPullingTime="5.272154144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:04:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:04:59.947785746 +0000 UTC m=+918.694975251" watchObservedRunningTime="2026-10-31 23:04:59.949024226 +0000 UTC m=+918.696213770"
Oct 31 23:05:02.545124 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:05:02.545083    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:05:08.927836 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:05:08.927803    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-64lft"
Oct 31 23:05:32.543901 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:05:32.543820    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:06:02.543639 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:06:02.543604    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:06:09.692301 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:09.692270    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:06:19.566529 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:19.566497    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6784f6bdbc-jgktm"]
Oct 31 23:06:19.569847 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:19.569829    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-jgktm"
Oct 31 23:06:19.573778 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:19.573754    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 23:06:19.576619 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:19.576317    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 31 23:06:19.576786 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:19.576765    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 23:06:19.576786 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:19.576779    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 23:06:19.581412 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:19.581387    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6784f6bdbc-jgktm"]
Oct 31 23:06:19.661618 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:19.661585    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06b93593-507a-4653-9fc8-59dacbdfd71f-maas-api-tls\") pod \"maas-api-6784f6bdbc-jgktm\" (UID: \"06b93593-507a-4653-9fc8-59dacbdfd71f\") " pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-jgktm"
Oct 31 23:06:19.661618 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:19.661625    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06b93593-507a-4653-9fc8-59dacbdfd71f-metrics-certs\") pod \"maas-api-6784f6bdbc-jgktm\" (UID: \"06b93593-507a-4653-9fc8-59dacbdfd71f\") " pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-jgktm"
Oct 31 23:06:19.661833 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:19.661651    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06b93593-507a-4653-9fc8-59dacbdfd71f-openshift-service-ca\") pod \"maas-api-6784f6bdbc-jgktm\" (UID: \"06b93593-507a-4653-9fc8-59dacbdfd71f\") " pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-jgktm"
Oct 31 23:06:19.661833 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:19.661736    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2btjt\" (UniqueName: \"kubernetes.io/projected/06b93593-507a-4653-9fc8-59dacbdfd71f-kube-api-access-2btjt\") pod \"maas-api-6784f6bdbc-jgktm\" (UID: \"06b93593-507a-4653-9fc8-59dacbdfd71f\") " pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-jgktm"
Oct 31 23:06:19.673904 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:19.673883    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d7j72\""
Oct 31 23:06:19.762124 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:19.762094    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06b93593-507a-4653-9fc8-59dacbdfd71f-maas-api-tls\") pod \"maas-api-6784f6bdbc-jgktm\" (UID: \"06b93593-507a-4653-9fc8-59dacbdfd71f\") " pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-jgktm"
Oct 31 23:06:19.762314 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:19.762131    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06b93593-507a-4653-9fc8-59dacbdfd71f-metrics-certs\") pod \"maas-api-6784f6bdbc-jgktm\" (UID: \"06b93593-507a-4653-9fc8-59dacbdfd71f\") " pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-jgktm"
Oct 31 23:06:19.762314 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:19.762156    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06b93593-507a-4653-9fc8-59dacbdfd71f-openshift-service-ca\") pod \"maas-api-6784f6bdbc-jgktm\" (UID: \"06b93593-507a-4653-9fc8-59dacbdfd71f\") " pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-jgktm"
Oct 31 23:06:19.762314 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:19.762192    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2btjt\" (UniqueName: \"kubernetes.io/projected/06b93593-507a-4653-9fc8-59dacbdfd71f-kube-api-access-2btjt\") pod \"maas-api-6784f6bdbc-jgktm\" (UID: \"06b93593-507a-4653-9fc8-59dacbdfd71f\") " pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-jgktm"
Oct 31 23:06:19.763136 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:19.763113    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06b93593-507a-4653-9fc8-59dacbdfd71f-openshift-service-ca\") pod \"maas-api-6784f6bdbc-jgktm\" (UID: \"06b93593-507a-4653-9fc8-59dacbdfd71f\") " pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-jgktm"
Oct 31 23:06:19.765280 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:19.765257    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06b93593-507a-4653-9fc8-59dacbdfd71f-maas-api-tls\") pod \"maas-api-6784f6bdbc-jgktm\" (UID: \"06b93593-507a-4653-9fc8-59dacbdfd71f\") " pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-jgktm"
Oct 31 23:06:19.765379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:19.765325    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06b93593-507a-4653-9fc8-59dacbdfd71f-metrics-certs\") pod \"maas-api-6784f6bdbc-jgktm\" (UID: \"06b93593-507a-4653-9fc8-59dacbdfd71f\") " pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-jgktm"
Oct 31 23:06:19.778408 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:19.778376    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2btjt\" (UniqueName: \"kubernetes.io/projected/06b93593-507a-4653-9fc8-59dacbdfd71f-kube-api-access-2btjt\") pod \"maas-api-6784f6bdbc-jgktm\" (UID: \"06b93593-507a-4653-9fc8-59dacbdfd71f\") " pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-jgktm"
Oct 31 23:06:19.882314 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:19.882276    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-jgktm"
Oct 31 23:06:20.156119 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:20.156043    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6784f6bdbc-jgktm"]
Oct 31 23:06:20.159853 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:06:20.159827    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06b93593_507a_4653_9fc8_59dacbdfd71f.slice/crio-c39f9a3c225ac31e61650a285e956e51c42116394b58924949a21a00a744d499 WatchSource:0}: Error finding container c39f9a3c225ac31e61650a285e956e51c42116394b58924949a21a00a744d499: Status 404 returned error can't find the container with id c39f9a3c225ac31e61650a285e956e51c42116394b58924949a21a00a744d499
Oct 31 23:06:20.234838 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:20.234806    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-jgktm" event={"ID":"06b93593-507a-4653-9fc8-59dacbdfd71f","Type":"ContainerStarted","Data":"c39f9a3c225ac31e61650a285e956e51c42116394b58924949a21a00a744d499"}
Oct 31 23:06:22.583857 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:22.583813    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-bjjv6"]
Oct 31 23:06:22.587236 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:22.587214    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-bjjv6"
Oct 31 23:06:22.599940 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:22.599907    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-bjjv6"]
Oct 31 23:06:22.681433 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:22.681397    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77ef2de4-0420-42ac-8b0f-eb7a117b7c4c-cert\") pod \"llmisvc-controller-manager-6466c554d8-bjjv6\" (UID: \"77ef2de4-0420-42ac-8b0f-eb7a117b7c4c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-bjjv6"
Oct 31 23:06:22.681625 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:22.681451    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xlx5\" (UniqueName: \"kubernetes.io/projected/77ef2de4-0420-42ac-8b0f-eb7a117b7c4c-kube-api-access-4xlx5\") pod \"llmisvc-controller-manager-6466c554d8-bjjv6\" (UID: \"77ef2de4-0420-42ac-8b0f-eb7a117b7c4c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-bjjv6"
Oct 31 23:06:22.782032 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:22.781990    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77ef2de4-0420-42ac-8b0f-eb7a117b7c4c-cert\") pod \"llmisvc-controller-manager-6466c554d8-bjjv6\" (UID: \"77ef2de4-0420-42ac-8b0f-eb7a117b7c4c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-bjjv6"
Oct 31 23:06:22.782227 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:22.782111    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xlx5\" (UniqueName: \"kubernetes.io/projected/77ef2de4-0420-42ac-8b0f-eb7a117b7c4c-kube-api-access-4xlx5\") pod \"llmisvc-controller-manager-6466c554d8-bjjv6\" (UID: \"77ef2de4-0420-42ac-8b0f-eb7a117b7c4c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-bjjv6"
Oct 31 23:06:22.784531 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:22.784500    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77ef2de4-0420-42ac-8b0f-eb7a117b7c4c-cert\") pod \"llmisvc-controller-manager-6466c554d8-bjjv6\" (UID: \"77ef2de4-0420-42ac-8b0f-eb7a117b7c4c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-bjjv6"
Oct 31 23:06:22.795775 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:22.795754    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xlx5\" (UniqueName: \"kubernetes.io/projected/77ef2de4-0420-42ac-8b0f-eb7a117b7c4c-kube-api-access-4xlx5\") pod \"llmisvc-controller-manager-6466c554d8-bjjv6\" (UID: \"77ef2de4-0420-42ac-8b0f-eb7a117b7c4c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-bjjv6"
Oct 31 23:06:22.904505 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:22.904471    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-bjjv6"
Oct 31 23:06:23.174600 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:23.174569    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-bjjv6"]
Oct 31 23:06:23.177510 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:06:23.177484    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77ef2de4_0420_42ac_8b0f_eb7a117b7c4c.slice/crio-1fea3a78deec5a8efeb60d75aa5774ae755607fa6a0d6b9040c4237132996964 WatchSource:0}: Error finding container 1fea3a78deec5a8efeb60d75aa5774ae755607fa6a0d6b9040c4237132996964: Status 404 returned error can't find the container with id 1fea3a78deec5a8efeb60d75aa5774ae755607fa6a0d6b9040c4237132996964
Oct 31 23:06:23.248437 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:23.248314    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-jgktm" event={"ID":"06b93593-507a-4653-9fc8-59dacbdfd71f","Type":"ContainerStarted","Data":"e5689157879eaca6989c1e8f95d302b3829f3cbf9fa23e6855c71cbb53719a2e"}
Oct 31 23:06:23.248437 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:23.248411    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-jgktm"
Oct 31 23:06:23.249506 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:23.249484    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-bjjv6" event={"ID":"77ef2de4-0420-42ac-8b0f-eb7a117b7c4c","Type":"ContainerStarted","Data":"1fea3a78deec5a8efeb60d75aa5774ae755607fa6a0d6b9040c4237132996964"}
Oct 31 23:06:23.267708 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:23.267662    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-jgktm" podStartSLOduration=2.068852898 podStartE2EDuration="4.267649285s" totalImagesPullingTime="2.198796387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:06:23.266889414 +0000 UTC m=+1002.014078918" watchObservedRunningTime="2026-10-31 23:06:23.267649285 +0000 UTC m=+1002.014838789"
Oct 31 23:06:24.255554 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:24.255511    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-bjjv6" event={"ID":"77ef2de4-0420-42ac-8b0f-eb7a117b7c4c","Type":"ContainerStarted","Data":"add4f30dbb9098b2f9b2b745b181ea0890653304e7d9b0751895ae03f0f5595c"}
Oct 31 23:06:24.255970 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:24.255766    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-bjjv6"
Oct 31 23:06:24.272049 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:24.272012    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-bjjv6" podStartSLOduration=1.870919506 podStartE2EDuration="2.271997172s" totalImagesPullingTime="401.077666ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:06:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:06:24.271895183 +0000 UTC m=+1003.019084687" watchObservedRunningTime="2026-10-31 23:06:24.271997172 +0000 UTC m=+1003.019186677"
Oct 31 23:06:29.260733 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:29.260701    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-jgktm"
Oct 31 23:06:32.544031 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:06:32.543988    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:06:55.261836 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:55.261765    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-bjjv6"
Oct 31 23:06:55.317974 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:55.317942    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-pmrkc"]
Oct 31 23:06:55.318215 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:55.318190    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-pmrkc" podUID="edcb76cb-2a03-451b-a920-209041f60ded" containerName="manager" containerID="cri-o://bfc8c92e9786720c901370a475a5c899a68e01b941cbad95e84acd807b766ee7" gracePeriod=30
Oct 31 23:06:55.556096 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:55.556068    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-pmrkc"
Oct 31 23:06:55.637642 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:55.637605    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edcb76cb-2a03-451b-a920-209041f60ded-cert\" (UniqueName: \"kubernetes.io/secret/edcb76cb-2a03-451b-a920-209041f60ded-cert\") pod \"edcb76cb-2a03-451b-a920-209041f60ded\" (UID: \"edcb76cb-2a03-451b-a920-209041f60ded\") "
Oct 31 23:06:55.637827 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:55.637706    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edcb76cb-2a03-451b-a920-209041f60ded-kube-api-access-t5bjx\" (UniqueName: \"kubernetes.io/projected/edcb76cb-2a03-451b-a920-209041f60ded-kube-api-access-t5bjx\") pod \"edcb76cb-2a03-451b-a920-209041f60ded\" (UID: \"edcb76cb-2a03-451b-a920-209041f60ded\") "
Oct 31 23:06:55.639731 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:55.639696    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edcb76cb-2a03-451b-a920-209041f60ded-kube-api-access-t5bjx" pod "edcb76cb-2a03-451b-a920-209041f60ded" (UID: "edcb76cb-2a03-451b-a920-209041f60ded"). InnerVolumeSpecName "kube-api-access-t5bjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:06:55.639844 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:55.639780    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edcb76cb-2a03-451b-a920-209041f60ded-cert" pod "edcb76cb-2a03-451b-a920-209041f60ded" (UID: "edcb76cb-2a03-451b-a920-209041f60ded"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:06:55.738393 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:55.738343    4301 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edcb76cb-2a03-451b-a920-209041f60ded-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:06:55.738393 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:55.738388    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5bjx\" (UniqueName: \"kubernetes.io/projected/edcb76cb-2a03-451b-a920-209041f60ded-kube-api-access-t5bjx\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:06:56.382870 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:56.382838    4301 generic.go:356] "Generic (PLEG): container finished" podID="edcb76cb-2a03-451b-a920-209041f60ded" containerID="bfc8c92e9786720c901370a475a5c899a68e01b941cbad95e84acd807b766ee7" exitCode=0
Oct 31 23:06:56.383276 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:56.382887    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-pmrkc"
Oct 31 23:06:56.383276 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:56.382915    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-pmrkc" event={"ID":"edcb76cb-2a03-451b-a920-209041f60ded","Type":"ContainerDied","Data":"bfc8c92e9786720c901370a475a5c899a68e01b941cbad95e84acd807b766ee7"}
Oct 31 23:06:56.383276 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:56.382957    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-pmrkc" event={"ID":"edcb76cb-2a03-451b-a920-209041f60ded","Type":"ContainerDied","Data":"67a0c8027e60b704e7e9024e945f5fd00ea78dd4a0e67562529829348b490fae"}
Oct 31 23:06:56.383276 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:56.382978    4301 scope.go:122] "RemoveContainer" containerID="bfc8c92e9786720c901370a475a5c899a68e01b941cbad95e84acd807b766ee7"
Oct 31 23:06:56.391510 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:56.391494    4301 scope.go:122] "RemoveContainer" containerID="bfc8c92e9786720c901370a475a5c899a68e01b941cbad95e84acd807b766ee7"
Oct 31 23:06:56.391723 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:06:56.391708    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfc8c92e9786720c901370a475a5c899a68e01b941cbad95e84acd807b766ee7\": container with ID starting with bfc8c92e9786720c901370a475a5c899a68e01b941cbad95e84acd807b766ee7 not found: ID does not exist" containerID="bfc8c92e9786720c901370a475a5c899a68e01b941cbad95e84acd807b766ee7"
Oct 31 23:06:56.391767 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:56.391734    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfc8c92e9786720c901370a475a5c899a68e01b941cbad95e84acd807b766ee7"} err="failed to get container status \"bfc8c92e9786720c901370a475a5c899a68e01b941cbad95e84acd807b766ee7\": rpc error: code = NotFound desc = could not find container \"bfc8c92e9786720c901370a475a5c899a68e01b941cbad95e84acd807b766ee7\": container with ID starting with bfc8c92e9786720c901370a475a5c899a68e01b941cbad95e84acd807b766ee7 not found: ID does not exist"
Oct 31 23:06:56.434189 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:56.434167    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-pmrkc"]
Oct 31 23:06:56.459325 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:56.459302    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-pmrkc"]
Oct 31 23:06:57.401271 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:06:57.401239    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edcb76cb-2a03-451b-a920-209041f60ded" path="/var/lib/kubelet/pods/edcb76cb-2a03-451b-a920-209041f60ded/volumes"
Oct 31 23:07:02.544541 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:07:02.544504    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:07:13.369777 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:13.369742    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:07:16.012473 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:16.012432    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:07:18.861816 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:18.861782    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:07:20.786017 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:20.785980    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:07:23.332056 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:23.331973    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:07:27.882647 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:27.882611    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:07:32.545718 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:07:32.545681    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:07:47.312670 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.312637    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"]
Oct 31 23:07:47.313232 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.313061    4301 cpu_manager.go:414] "containerMap: removing container" podUID="edcb76cb-2a03-451b-a920-209041f60ded" containerName="manager"
Oct 31 23:07:47.313232 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.313081    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edcb76cb-2a03-451b-a920-209041f60ded" containerName="manager"
Oct 31 23:07:47.313232 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.313153    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="edcb76cb-2a03-451b-a920-209041f60ded" containerName="manager"
Oct 31 23:07:47.317008 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.316988    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:07:47.346784 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.346726    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"]
Oct 31 23:07:47.355149 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.355126    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-v559f\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:07:47.355285 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.355155    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-v559f\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:07:47.355285 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.355187    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86a238da-6602-4f4b-a300-be4347d907e5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-v559f\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:07:47.355375 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.355292    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-v559f\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:07:47.355375 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.355338    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-v559f\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:07:47.355452 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.355407    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khh2n\" (UniqueName: \"kubernetes.io/projected/86a238da-6602-4f4b-a300-be4347d907e5-kube-api-access-khh2n\") pod \"maas-default-gateway-openshift-default-6886b96f9-v559f\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:07:47.355487 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.355455    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86a238da-6602-4f4b-a300-be4347d907e5-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-v559f\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:07:47.355547 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.355524    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-v559f\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:07:47.355606 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.355591    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86a238da-6602-4f4b-a300-be4347d907e5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-v559f\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:07:47.405971 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.405945    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-nf6c6\""
Oct 31 23:07:47.456907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.456871    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86a238da-6602-4f4b-a300-be4347d907e5-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-v559f\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:07:47.456907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.456910    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-v559f\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:07:47.457088 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.457013    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86a238da-6602-4f4b-a300-be4347d907e5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-v559f\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:07:47.457088 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.457044    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-v559f\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:07:47.457088 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.457062    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-v559f\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:07:47.457088 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.457079    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86a238da-6602-4f4b-a300-be4347d907e5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-v559f\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:07:47.457281 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.457095    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-v559f\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:07:47.457281 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.457111    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-v559f\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:07:47.457281 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.457133    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khh2n\" (UniqueName: \"kubernetes.io/projected/86a238da-6602-4f4b-a300-be4347d907e5-kube-api-access-khh2n\") pod \"maas-default-gateway-openshift-default-6886b96f9-v559f\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:07:47.457562 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.457460    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-v559f\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:07:47.457562 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.457514    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-v559f\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:07:47.457753 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.457719    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-v559f\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:07:47.457873 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.457817    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-v559f\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:07:47.457998 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.457979    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86a238da-6602-4f4b-a300-be4347d907e5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-v559f\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:07:47.459320 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.459299    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-v559f\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:07:47.459664 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.459642    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86a238da-6602-4f4b-a300-be4347d907e5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-v559f\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:07:47.471697 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.471675    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86a238da-6602-4f4b-a300-be4347d907e5-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-v559f\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:07:47.472144 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.472127    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khh2n\" (UniqueName: \"kubernetes.io/projected/86a238da-6602-4f4b-a300-be4347d907e5-kube-api-access-khh2n\") pod \"maas-default-gateway-openshift-default-6886b96f9-v559f\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:07:47.629626 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.629602    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:07:47.909780 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:47.909693    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"]
Oct 31 23:07:47.913553 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:07:47.913528    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a238da_6602_4f4b_a300_be4347d907e5.slice/crio-5922f22e070b4de6f675a085e3fe4631a0b855679b05ab229ea0026f38b7bb9b WatchSource:0}: Error finding container 5922f22e070b4de6f675a085e3fe4631a0b855679b05ab229ea0026f38b7bb9b: Status 404 returned error can't find the container with id 5922f22e070b4de6f675a085e3fe4631a0b855679b05ab229ea0026f38b7bb9b
Oct 31 23:07:48.592111 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:48.592075    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f" event={"ID":"86a238da-6602-4f4b-a300-be4347d907e5","Type":"ContainerStarted","Data":"5922f22e070b4de6f675a085e3fe4631a0b855679b05ab229ea0026f38b7bb9b"}
Oct 31 23:07:50.804565 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:50.804534    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:07:50.804827 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:50.804603    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:07:50.804827 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:50.804636    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:07:51.606242 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:51.606207    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f" event={"ID":"86a238da-6602-4f4b-a300-be4347d907e5","Type":"ContainerStarted","Data":"f80ca256dc927e3772f9e02f43d640370011428d71d9a44d4d3e155ded135e57"}
Oct 31 23:07:51.650401 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:51.636751    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:07:51.654987 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:51.654952    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-v559f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 31 23:07:51.655129 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:51.655001    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f" podUID="86a238da-6602-4f4b-a300-be4347d907e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 31 23:07:51.679561 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:51.679507    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f" podStartSLOduration=1.79130403 podStartE2EDuration="4.679490455s" totalImagesPullingTime="2.888186425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:07:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:07:51.677564128 +0000 UTC m=+1090.424753631" watchObservedRunningTime="2026-10-31 23:07:51.679490455 +0000 UTC m=+1090.426679961"
Oct 31 23:07:52.649382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:52.645769    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-v559f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 31 23:07:52.649382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:52.645833    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f" podUID="86a238da-6602-4f4b-a300-be4347d907e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 31 23:07:53.636398 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:53.633755    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-v559f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 31 23:07:53.636398 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:53.633814    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f" podUID="86a238da-6602-4f4b-a300-be4347d907e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 31 23:07:54.651385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:54.649828    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-v559f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 31 23:07:54.651385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:54.649890    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f" podUID="86a238da-6602-4f4b-a300-be4347d907e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 31 23:07:55.650384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:55.648793    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-v559f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 31 23:07:55.650384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:55.648872    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f" podUID="86a238da-6602-4f4b-a300-be4347d907e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 31 23:07:56.635742 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:56.635692    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-v559f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 31 23:07:56.636244 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:56.635771    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f" podUID="86a238da-6602-4f4b-a300-be4347d907e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 31 23:07:57.630388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:57.630345    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-v559f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 31 23:07:57.630564 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:57.630416    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f" podUID="86a238da-6602-4f4b-a300-be4347d907e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 31 23:07:58.915182 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:58.915149    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:07:58.915875 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:58.915844    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:07:58.916810 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:07:58.916791    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:08:02.285820 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.285789    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"]
Oct 31 23:08:02.289107 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.289075    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:08:02.305622 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.305598    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"]
Oct 31 23:08:02.408995 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.408964    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-922mr\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:08:02.408995 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.408999    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-922mr\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:08:02.409198 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.409035    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98hl9\" (UniqueName: \"kubernetes.io/projected/14a4227c-cbbd-4193-81e9-56287613d98b-kube-api-access-98hl9\") pod \"maas-default-gateway-openshift-default-6886b96f9-922mr\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:08:02.409198 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.409152    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-922mr\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:08:02.409198 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.409181    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14a4227c-cbbd-4193-81e9-56287613d98b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-922mr\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:08:02.409293 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.409210    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14a4227c-cbbd-4193-81e9-56287613d98b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-922mr\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:08:02.409293 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.409270    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-922mr\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:08:02.409374 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.409302    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14a4227c-cbbd-4193-81e9-56287613d98b-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-922mr\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:08:02.409374 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.409352    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-922mr\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:08:02.510262 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.510229    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-922mr\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:08:02.510262 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.510259    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-922mr\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:08:02.510524 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.510288    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98hl9\" (UniqueName: \"kubernetes.io/projected/14a4227c-cbbd-4193-81e9-56287613d98b-kube-api-access-98hl9\") pod \"maas-default-gateway-openshift-default-6886b96f9-922mr\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:08:02.510524 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.510327    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-922mr\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:08:02.510524 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.510353    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14a4227c-cbbd-4193-81e9-56287613d98b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-922mr\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:08:02.510524 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.510409    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14a4227c-cbbd-4193-81e9-56287613d98b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-922mr\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:08:02.510524 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.510437    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-922mr\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:08:02.510524 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.510460    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14a4227c-cbbd-4193-81e9-56287613d98b-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-922mr\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:08:02.510524 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.510499    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-922mr\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:08:02.510864 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.510668    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-922mr\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:08:02.510864 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.510786    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-922mr\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:08:02.510864 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.510843    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-922mr\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:08:02.510969 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.510938    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-922mr\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:08:02.511324 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.511305    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14a4227c-cbbd-4193-81e9-56287613d98b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-922mr\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:08:02.512603 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.512584    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-922mr\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:08:02.512934 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.512911    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14a4227c-cbbd-4193-81e9-56287613d98b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-922mr\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:08:02.521842 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.521815    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14a4227c-cbbd-4193-81e9-56287613d98b-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-922mr\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:08:02.521956 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.521937    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98hl9\" (UniqueName: \"kubernetes.io/projected/14a4227c-cbbd-4193-81e9-56287613d98b-kube-api-access-98hl9\") pod \"maas-default-gateway-openshift-default-6886b96f9-922mr\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:08:02.544242 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:08:02.544169    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:08:02.602623 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.602592    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:08:02.884685 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.884659    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"]
Oct 31 23:08:02.887384 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:08:02.887334    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a4227c_cbbd_4193_81e9_56287613d98b.slice/crio-fd884226d6e3e24008d437fbbaf8b23338a3bce3152c7d520bc883e48ec4c4a2 WatchSource:0}: Error finding container fd884226d6e3e24008d437fbbaf8b23338a3bce3152c7d520bc883e48ec4c4a2: Status 404 returned error can't find the container with id fd884226d6e3e24008d437fbbaf8b23338a3bce3152c7d520bc883e48ec4c4a2
Oct 31 23:08:02.890233 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.890201    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:08:02.890330 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.890265    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:08:02.890330 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:02.890293    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:08:03.659070 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:03.659035    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr" event={"ID":"14a4227c-cbbd-4193-81e9-56287613d98b","Type":"ContainerStarted","Data":"dbb43a7620a5ec4d93fc790a8ebaa7149fb982445c0e42082db6b9eb7e89542b"}
Oct 31 23:08:03.659570 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:03.659099    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr" event={"ID":"14a4227c-cbbd-4193-81e9-56287613d98b","Type":"ContainerStarted","Data":"fd884226d6e3e24008d437fbbaf8b23338a3bce3152c7d520bc883e48ec4c4a2"}
Oct 31 23:08:03.681266 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:03.681212    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr" podStartSLOduration=1.681196954 podStartE2EDuration="1.681196954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:03.680539594 +0000 UTC m=+1102.427729102" watchObservedRunningTime="2026-10-31 23:08:03.681196954 +0000 UTC m=+1102.428386460"
Oct 31 23:08:04.603682 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:04.603645    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:08:04.604968 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:04.604939    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-922mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 31 23:08:04.605103 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:04.604997    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr" podUID="14a4227c-cbbd-4193-81e9-56287613d98b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 31 23:08:05.603756 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:05.603715    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-922mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 31 23:08:05.604148 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:05.603776    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr" podUID="14a4227c-cbbd-4193-81e9-56287613d98b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 31 23:08:06.603434 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:06.603399    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-922mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 31 23:08:06.603608 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:06.603450    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr" podUID="14a4227c-cbbd-4193-81e9-56287613d98b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 31 23:08:07.603862 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:07.603828    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-922mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 31 23:08:07.604242 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:07.603877    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr" podUID="14a4227c-cbbd-4193-81e9-56287613d98b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 31 23:08:08.603535 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:08.603495    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-922mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 31 23:08:08.603720 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:08.603550    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr" podUID="14a4227c-cbbd-4193-81e9-56287613d98b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 31 23:08:09.603909 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:09.603872    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-922mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 31 23:08:09.604432 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:09.603931    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr" podUID="14a4227c-cbbd-4193-81e9-56287613d98b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 31 23:08:10.943008 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:10.942975    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:08:10.944087 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:10.944061    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:08:10.944988 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:10.944964    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:08:17.301638 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.301600    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"]
Oct 31 23:08:17.305321 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.305301    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:08:17.319620 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.319598    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"]
Oct 31 23:08:17.347158 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.347127    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lprvg\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:08:17.347320 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.347177    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lprvg\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:08:17.347320 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.347244    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pttg\" (UniqueName: \"kubernetes.io/projected/201d729f-baac-4eb9-bcbc-497ad72370b9-kube-api-access-5pttg\") pod \"maas-default-gateway-openshift-default-6886b96f9-lprvg\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:08:17.347459 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.347338    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-lprvg\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:08:17.347459 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.347405    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/201d729f-baac-4eb9-bcbc-497ad72370b9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-lprvg\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:08:17.347459 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.347427    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-lprvg\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:08:17.347459 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.347446    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-lprvg\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:08:17.347611 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.347560    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/201d729f-baac-4eb9-bcbc-497ad72370b9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-lprvg\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:08:17.347648 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.347636    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/201d729f-baac-4eb9-bcbc-497ad72370b9-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-lprvg\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:08:17.448557 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.448515    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lprvg\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:08:17.448557 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.448560    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lprvg\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:08:17.448805 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.448584    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pttg\" (UniqueName: \"kubernetes.io/projected/201d729f-baac-4eb9-bcbc-497ad72370b9-kube-api-access-5pttg\") pod \"maas-default-gateway-openshift-default-6886b96f9-lprvg\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:08:17.448805 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.448626    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-lprvg\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:08:17.448805 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.448670    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/201d729f-baac-4eb9-bcbc-497ad72370b9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-lprvg\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:08:17.448805 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.448700    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-lprvg\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:08:17.448805 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.448731    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-lprvg\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:08:17.448805 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.448775    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/201d729f-baac-4eb9-bcbc-497ad72370b9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-lprvg\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:08:17.449175 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.448817    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/201d729f-baac-4eb9-bcbc-497ad72370b9-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-lprvg\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:08:17.449175 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.449047    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lprvg\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:08:17.449310 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.449248    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-lprvg\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:08:17.449459 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.449431    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lprvg\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:08:17.449716 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.449460    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-lprvg\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:08:17.449817 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.449792    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/201d729f-baac-4eb9-bcbc-497ad72370b9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-lprvg\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:08:17.451137 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.451115    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-lprvg\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:08:17.451242 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.451219    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/201d729f-baac-4eb9-bcbc-497ad72370b9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-lprvg\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:08:17.460343 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.460312    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/201d729f-baac-4eb9-bcbc-497ad72370b9-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-lprvg\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:08:17.460480 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.460454    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pttg\" (UniqueName: \"kubernetes.io/projected/201d729f-baac-4eb9-bcbc-497ad72370b9-kube-api-access-5pttg\") pod \"maas-default-gateway-openshift-default-6886b96f9-lprvg\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:08:17.617938 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.617857    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:08:17.899900 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.899818    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"]
Oct 31 23:08:17.902737 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:08:17.902710    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod201d729f_baac_4eb9_bcbc_497ad72370b9.slice/crio-1780aa8a4725810f417f580f773c1f88d295f267242e83128492cdf15d3e3ec7 WatchSource:0}: Error finding container 1780aa8a4725810f417f580f773c1f88d295f267242e83128492cdf15d3e3ec7: Status 404 returned error can't find the container with id 1780aa8a4725810f417f580f773c1f88d295f267242e83128492cdf15d3e3ec7
Oct 31 23:08:17.904769 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.904738    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:08:17.904854 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.904803    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:08:17.904854 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:17.904832    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:08:18.721076 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:18.721038    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg" event={"ID":"201d729f-baac-4eb9-bcbc-497ad72370b9","Type":"ContainerStarted","Data":"b5ca80e09dddaf0f88f881e951e08a88b79217f5d0df1e2ca5a4279cf7231c71"}
Oct 31 23:08:18.721535 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:18.721087    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg" event={"ID":"201d729f-baac-4eb9-bcbc-497ad72370b9","Type":"ContainerStarted","Data":"1780aa8a4725810f417f580f773c1f88d295f267242e83128492cdf15d3e3ec7"}
Oct 31 23:08:19.618951 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:19.618914    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:08:19.620081 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:19.620062    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lprvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 31 23:08:19.620138 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:19.620101    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg" podUID="201d729f-baac-4eb9-bcbc-497ad72370b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 31 23:08:20.618476 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:20.618430    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lprvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 31 23:08:20.618963 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:20.618489    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg" podUID="201d729f-baac-4eb9-bcbc-497ad72370b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 31 23:08:21.620051 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:21.620018    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lprvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 31 23:08:21.620550 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:21.620507    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg" podUID="201d729f-baac-4eb9-bcbc-497ad72370b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 31 23:08:22.618657 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:22.618621    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lprvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 31 23:08:22.618828 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:22.618673    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg" podUID="201d729f-baac-4eb9-bcbc-497ad72370b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 31 23:08:23.619090 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:23.619056    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lprvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 31 23:08:23.619485 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:23.619120    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg" podUID="201d729f-baac-4eb9-bcbc-497ad72370b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 31 23:08:24.618328 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:24.618289    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lprvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 31 23:08:24.618527 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:24.618353    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg" podUID="201d729f-baac-4eb9-bcbc-497ad72370b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 31 23:08:25.923964 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:25.923931    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:08:25.924477 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:25.924333    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:08:25.925084 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:25.925066    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:08:25.958300 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:25.958248    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg" podStartSLOduration=8.958235002 podStartE2EDuration="8.958235002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:18.748294056 +0000 UTC m=+1117.495483562" watchObservedRunningTime="2026-10-31 23:08:25.958235002 +0000 UTC m=+1124.705424507"
Oct 31 23:08:32.544871 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:08:32.544831    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:08:39.430460 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:39.430426    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"]
Oct 31 23:08:39.434047 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:39.434026    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:08:39.437512 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:39.437491    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-7d7eed-openshift-default-dockercfg-lp7z5\""
Oct 31 23:08:39.451382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:39.451335    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"]
Oct 31 23:08:39.543115 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:39.543075    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-credential-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:08:39.543115 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:39.543109    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-istio-data\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:08:39.543389 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:39.543132    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-workload-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:08:39.543389 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:39.543204    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-istio-envoy\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:08:39.543389 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:39.543274    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0195c12-01bd-4f4a-a72a-b608150516f2-istio-podinfo\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:08:39.543554 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:39.543406    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-workload-certs\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:08:39.543554 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:39.543430    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0195c12-01bd-4f4a-a72a-b608150516f2-istio-token\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:08:39.543554 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:39.543448    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdbff\" (UniqueName: \"kubernetes.io/projected/d0195c12-01bd-4f4a-a72a-b608150516f2-kube-api-access-pdbff\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:08:39.543554 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:39.543492    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0195c12-01bd-4f4a-a72a-b608150516f2-istiod-ca-cert\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:08:39.644195 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:39.644153    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0195c12-01bd-4f4a-a72a-b608150516f2-istiod-ca-cert\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:08:39.644533 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:39.644511    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-credential-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:08:39.644675 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:39.644659    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-istio-data\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:08:39.644798 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:39.644782    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-workload-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:08:39.644901 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:39.644887    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-istio-envoy\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:08:39.645053 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:39.645040    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0195c12-01bd-4f4a-a72a-b608150516f2-istio-podinfo\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:08:39.645223 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:39.645204    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-workload-certs\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:08:39.645350 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:39.645333    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0195c12-01bd-4f4a-a72a-b608150516f2-istio-token\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:08:39.645532 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:39.645510    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdbff\" (UniqueName: \"kubernetes.io/projected/d0195c12-01bd-4f4a-a72a-b608150516f2-kube-api-access-pdbff\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:08:39.645749 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:39.645720    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-workload-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:08:39.647207 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:39.646285    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-credential-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:08:39.650697 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:39.650649    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-istio-envoy\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:08:39.650803 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:39.650713    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-workload-certs\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:08:39.650923 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:39.650904    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0195c12-01bd-4f4a-a72a-b608150516f2-istiod-ca-cert\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:08:39.651061 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:39.651038    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0195c12-01bd-4f4a-a72a-b608150516f2-istio-podinfo\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:08:39.651343 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:39.651316    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-istio-data\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:08:39.660168 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:39.660120    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0195c12-01bd-4f4a-a72a-b608150516f2-istio-token\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:08:39.660280 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:39.660258    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdbff\" (UniqueName: \"kubernetes.io/projected/d0195c12-01bd-4f4a-a72a-b608150516f2-kube-api-access-pdbff\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:08:39.759931 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:39.759853    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:08:40.050997 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:40.050966    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"]
Oct 31 23:08:40.053886 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:08:40.053854    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0195c12_01bd_4f4a_a72a_b608150516f2.slice/crio-3fa7ae01589f39a9f78add93664f5b1ebf904cc781b0a3a59a0a92ba6db80331 WatchSource:0}: Error finding container 3fa7ae01589f39a9f78add93664f5b1ebf904cc781b0a3a59a0a92ba6db80331: Status 404 returned error can't find the container with id 3fa7ae01589f39a9f78add93664f5b1ebf904cc781b0a3a59a0a92ba6db80331
Oct 31 23:08:40.056799 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:40.056757    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:08:40.056939 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:40.056851    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:08:40.056939 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:40.056891    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:08:40.836275 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:40.836235    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg" event={"ID":"d0195c12-01bd-4f4a-a72a-b608150516f2","Type":"ContainerStarted","Data":"ef96b32c0a70a58bf905ba846040660f49d77848cc1a7acb6c6e6567c960e7b5"}
Oct 31 23:08:40.836715 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:40.836290    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg" event={"ID":"d0195c12-01bd-4f4a-a72a-b608150516f2","Type":"ContainerStarted","Data":"3fa7ae01589f39a9f78add93664f5b1ebf904cc781b0a3a59a0a92ba6db80331"}
Oct 31 23:08:40.859257 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:40.859212    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg" podStartSLOduration=1.859197524 podStartE2EDuration="1.859197524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:40.857421809 +0000 UTC m=+1139.604611325" watchObservedRunningTime="2026-10-31 23:08:40.859197524 +0000 UTC m=+1139.606387029"
Oct 31 23:08:41.761068 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:41.761027    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:08:41.766557 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:41.766528    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:08:41.842062 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:41.842033    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:08:41.843041 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:41.843023    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:08:47.994752 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:47.994709    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:08:48.188089 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:48.188060    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7d7eed-7c97f4ff9f-whnrt"]
Oct 31 23:08:48.192649 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:48.192631    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d7eed-7c97f4ff9f-whnrt"
Oct 31 23:08:48.199155 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:48.199080    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-7d7eed-dockercfg-4f64k\""
Oct 31 23:08:48.199287 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:48.199269    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-7d7eed\""
Oct 31 23:08:48.206707 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:48.206685    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7d7eed-7c97f4ff9f-whnrt"]
Oct 31 23:08:48.334886 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:48.334800    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq6g6\" (UniqueName: \"kubernetes.io/projected/a3bc4bb3-e967-4fe3-a6b9-71377f8f6047-kube-api-access-zq6g6\") pod \"payload-pre-processing-e2e-worker-w6-7d7eed-7c97f4ff9f-whnrt\" (UID: \"a3bc4bb3-e967-4fe3-a6b9-71377f8f6047\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d7eed-7c97f4ff9f-whnrt"
Oct 31 23:08:48.335061 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:48.334977    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3bc4bb3-e967-4fe3-a6b9-71377f8f6047-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-7d7eed-7c97f4ff9f-whnrt\" (UID: \"a3bc4bb3-e967-4fe3-a6b9-71377f8f6047\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d7eed-7c97f4ff9f-whnrt"
Oct 31 23:08:48.440907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:48.440864    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq6g6\" (UniqueName: \"kubernetes.io/projected/a3bc4bb3-e967-4fe3-a6b9-71377f8f6047-kube-api-access-zq6g6\") pod \"payload-pre-processing-e2e-worker-w6-7d7eed-7c97f4ff9f-whnrt\" (UID: \"a3bc4bb3-e967-4fe3-a6b9-71377f8f6047\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d7eed-7c97f4ff9f-whnrt"
Oct 31 23:08:48.441258 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:48.441237    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3bc4bb3-e967-4fe3-a6b9-71377f8f6047-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-7d7eed-7c97f4ff9f-whnrt\" (UID: \"a3bc4bb3-e967-4fe3-a6b9-71377f8f6047\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d7eed-7c97f4ff9f-whnrt"
Oct 31 23:08:48.442278 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:48.442252    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3bc4bb3-e967-4fe3-a6b9-71377f8f6047-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-7d7eed-7c97f4ff9f-whnrt\" (UID: \"a3bc4bb3-e967-4fe3-a6b9-71377f8f6047\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d7eed-7c97f4ff9f-whnrt"
Oct 31 23:08:48.464610 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:48.464570    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq6g6\" (UniqueName: \"kubernetes.io/projected/a3bc4bb3-e967-4fe3-a6b9-71377f8f6047-kube-api-access-zq6g6\") pod \"payload-pre-processing-e2e-worker-w6-7d7eed-7c97f4ff9f-whnrt\" (UID: \"a3bc4bb3-e967-4fe3-a6b9-71377f8f6047\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d7eed-7c97f4ff9f-whnrt"
Oct 31 23:08:48.505455 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:48.505413    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d7eed-7c97f4ff9f-whnrt"
Oct 31 23:08:48.778903 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:48.778865    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7d7eed-7c97f4ff9f-whnrt"]
Oct 31 23:08:48.781788 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:08:48.781740    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3bc4bb3_e967_4fe3_a6b9_71377f8f6047.slice/crio-a61a1b936e354a3bbfb65ad2d30566b13b125aa90e9bb4233c025fd540d748f5 WatchSource:0}: Error finding container a61a1b936e354a3bbfb65ad2d30566b13b125aa90e9bb4233c025fd540d748f5: Status 404 returned error can't find the container with id a61a1b936e354a3bbfb65ad2d30566b13b125aa90e9bb4233c025fd540d748f5
Oct 31 23:08:48.879964 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:48.879930    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d7eed-7c97f4ff9f-whnrt" event={"ID":"a3bc4bb3-e967-4fe3-a6b9-71377f8f6047","Type":"ContainerStarted","Data":"a61a1b936e354a3bbfb65ad2d30566b13b125aa90e9bb4233c025fd540d748f5"}
Oct 31 23:08:50.117847 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:50.117815    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr"]
Oct 31 23:08:50.124670 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:50.124648    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr"
Oct 31 23:08:50.133382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:50.133324    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-67fce9-serving-cert\""
Oct 31 23:08:50.136975 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:50.136950    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr"]
Oct 31 23:08:50.263100 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:50.263071    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aac5700-d634-407f-98f1-e7ab6beeebc8-maas-api-tls\") pod \"maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr\" (UID: \"9aac5700-d634-407f-98f1-e7ab6beeebc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr"
Oct 31 23:08:50.263254 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:50.263176    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aac5700-d634-407f-98f1-e7ab6beeebc8-metrics-certs\") pod \"maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr\" (UID: \"9aac5700-d634-407f-98f1-e7ab6beeebc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr"
Oct 31 23:08:50.263318 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:50.263270    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aac5700-d634-407f-98f1-e7ab6beeebc8-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr\" (UID: \"9aac5700-d634-407f-98f1-e7ab6beeebc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr"
Oct 31 23:08:50.263382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:50.263345    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kzkr\" (UniqueName: \"kubernetes.io/projected/9aac5700-d634-407f-98f1-e7ab6beeebc8-kube-api-access-6kzkr\") pod \"maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr\" (UID: \"9aac5700-d634-407f-98f1-e7ab6beeebc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr"
Oct 31 23:08:50.364826 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:50.364793    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aac5700-d634-407f-98f1-e7ab6beeebc8-metrics-certs\") pod \"maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr\" (UID: \"9aac5700-d634-407f-98f1-e7ab6beeebc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr"
Oct 31 23:08:50.365029 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:50.364900    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aac5700-d634-407f-98f1-e7ab6beeebc8-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr\" (UID: \"9aac5700-d634-407f-98f1-e7ab6beeebc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr"
Oct 31 23:08:50.365029 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:50.364933    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kzkr\" (UniqueName: \"kubernetes.io/projected/9aac5700-d634-407f-98f1-e7ab6beeebc8-kube-api-access-6kzkr\") pod \"maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr\" (UID: \"9aac5700-d634-407f-98f1-e7ab6beeebc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr"
Oct 31 23:08:50.365029 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:50.364968    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aac5700-d634-407f-98f1-e7ab6beeebc8-maas-api-tls\") pod \"maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr\" (UID: \"9aac5700-d634-407f-98f1-e7ab6beeebc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr"
Oct 31 23:08:50.365998 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:50.365971    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aac5700-d634-407f-98f1-e7ab6beeebc8-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr\" (UID: \"9aac5700-d634-407f-98f1-e7ab6beeebc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr"
Oct 31 23:08:50.367643 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:50.367621    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aac5700-d634-407f-98f1-e7ab6beeebc8-metrics-certs\") pod \"maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr\" (UID: \"9aac5700-d634-407f-98f1-e7ab6beeebc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr"
Oct 31 23:08:50.367744 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:50.367684    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aac5700-d634-407f-98f1-e7ab6beeebc8-maas-api-tls\") pod \"maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr\" (UID: \"9aac5700-d634-407f-98f1-e7ab6beeebc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr"
Oct 31 23:08:50.377807 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:50.377782    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kzkr\" (UniqueName: \"kubernetes.io/projected/9aac5700-d634-407f-98f1-e7ab6beeebc8-kube-api-access-6kzkr\") pod \"maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr\" (UID: \"9aac5700-d634-407f-98f1-e7ab6beeebc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr"
Oct 31 23:08:50.442210 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:50.442176    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr"
Oct 31 23:08:50.754414 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:50.754336    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr"]
Oct 31 23:08:50.757946 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:08:50.757919    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aac5700_d634_407f_98f1_e7ab6beeebc8.slice/crio-d2e65365cd42777a657401a164b02f846fafb9f6d94582ce3ad012be0e8bb9bd WatchSource:0}: Error finding container d2e65365cd42777a657401a164b02f846fafb9f6d94582ce3ad012be0e8bb9bd: Status 404 returned error can't find the container with id d2e65365cd42777a657401a164b02f846fafb9f6d94582ce3ad012be0e8bb9bd
Oct 31 23:08:50.893330 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:50.893291    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr" event={"ID":"9aac5700-d634-407f-98f1-e7ab6beeebc8","Type":"ContainerStarted","Data":"d2e65365cd42777a657401a164b02f846fafb9f6d94582ce3ad012be0e8bb9bd"}
Oct 31 23:08:51.707370 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:51.707317    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg"]
Oct 31 23:08:51.711535 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:51.711509    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg"
Oct 31 23:08:51.719066 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:51.719043    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-7d7eed-serving-cert\""
Oct 31 23:08:51.727210 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:51.727182    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg"]
Oct 31 23:08:51.778620 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:51.778586    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ee9b5ef-cded-4849-8ebc-86c305c4e8a7-maas-api-tls\") pod \"maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg\" (UID: \"7ee9b5ef-cded-4849-8ebc-86c305c4e8a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg"
Oct 31 23:08:51.778792 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:51.778757    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7pdn\" (UniqueName: \"kubernetes.io/projected/7ee9b5ef-cded-4849-8ebc-86c305c4e8a7-kube-api-access-t7pdn\") pod \"maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg\" (UID: \"7ee9b5ef-cded-4849-8ebc-86c305c4e8a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg"
Oct 31 23:08:51.778843 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:51.778798    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ee9b5ef-cded-4849-8ebc-86c305c4e8a7-metrics-certs\") pod \"maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg\" (UID: \"7ee9b5ef-cded-4849-8ebc-86c305c4e8a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg"
Oct 31 23:08:51.778885 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:51.778833    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee9b5ef-cded-4849-8ebc-86c305c4e8a7-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg\" (UID: \"7ee9b5ef-cded-4849-8ebc-86c305c4e8a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg"
Oct 31 23:08:51.879271 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:51.879232    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7pdn\" (UniqueName: \"kubernetes.io/projected/7ee9b5ef-cded-4849-8ebc-86c305c4e8a7-kube-api-access-t7pdn\") pod \"maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg\" (UID: \"7ee9b5ef-cded-4849-8ebc-86c305c4e8a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg"
Oct 31 23:08:51.879271 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:51.879272    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ee9b5ef-cded-4849-8ebc-86c305c4e8a7-metrics-certs\") pod \"maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg\" (UID: \"7ee9b5ef-cded-4849-8ebc-86c305c4e8a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg"
Oct 31 23:08:51.879550 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:51.879302    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee9b5ef-cded-4849-8ebc-86c305c4e8a7-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg\" (UID: \"7ee9b5ef-cded-4849-8ebc-86c305c4e8a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg"
Oct 31 23:08:51.879550 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:51.879437    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ee9b5ef-cded-4849-8ebc-86c305c4e8a7-maas-api-tls\") pod \"maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg\" (UID: \"7ee9b5ef-cded-4849-8ebc-86c305c4e8a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg"
Oct 31 23:08:51.879658 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:08:51.879563    4301 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d7eed-serving-cert: secret "maas-api-e2e-worker-w6-7d7eed-serving-cert" not found
Oct 31 23:08:51.879658 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:08:51.879649    4301 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ee9b5ef-cded-4849-8ebc-86c305c4e8a7-maas-api-tls podName:7ee9b5ef-cded-4849-8ebc-86c305c4e8a7 nodeName:}" failed. No retries permitted until 2026-10-31 23:08:52.379631088 +0000 UTC m=+1151.126820576 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7ee9b5ef-cded-4849-8ebc-86c305c4e8a7-maas-api-tls") pod "maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg" (UID: "7ee9b5ef-cded-4849-8ebc-86c305c4e8a7") : secret "maas-api-e2e-worker-w6-7d7eed-serving-cert" not found
Oct 31 23:08:51.880059 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:51.880042    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee9b5ef-cded-4849-8ebc-86c305c4e8a7-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg\" (UID: \"7ee9b5ef-cded-4849-8ebc-86c305c4e8a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg"
Oct 31 23:08:51.881740 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:51.881719    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ee9b5ef-cded-4849-8ebc-86c305c4e8a7-metrics-certs\") pod \"maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg\" (UID: \"7ee9b5ef-cded-4849-8ebc-86c305c4e8a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg"
Oct 31 23:08:51.893068 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:51.893040    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7pdn\" (UniqueName: \"kubernetes.io/projected/7ee9b5ef-cded-4849-8ebc-86c305c4e8a7-kube-api-access-t7pdn\") pod \"maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg\" (UID: \"7ee9b5ef-cded-4849-8ebc-86c305c4e8a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg"
Oct 31 23:08:51.899022 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:51.898993    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d7eed-7c97f4ff9f-whnrt" event={"ID":"a3bc4bb3-e967-4fe3-a6b9-71377f8f6047","Type":"ContainerStarted","Data":"5de4cae8fa83ea1bedeac765e1e1ae9941651750c3419eadbef02ad843453b20"}
Oct 31 23:08:51.899172 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:51.899138    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d7eed-7c97f4ff9f-whnrt"
Oct 31 23:08:51.900883 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:51.900863    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr" event={"ID":"9aac5700-d634-407f-98f1-e7ab6beeebc8","Type":"ContainerStarted","Data":"70018d80bdc77c29ee2bae65e41b87b7300760a142c944a0c6f74f96c1f5b1c6"}
Oct 31 23:08:51.900999 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:51.900988    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr"
Oct 31 23:08:51.916221 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:51.916185    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d7eed-7c97f4ff9f-whnrt" podStartSLOduration=1.634896444 podStartE2EDuration="3.916173213s" totalImagesPullingTime="2.281276769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:51.91477864 +0000 UTC m=+1150.661968144" watchObservedRunningTime="2026-10-31 23:08:51.916173213 +0000 UTC m=+1150.663362717"
Oct 31 23:08:51.933857 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:51.933799    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr" podStartSLOduration=1.520772901 podStartE2EDuration="1.933783017s" totalImagesPullingTime="413.010116ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:51.932342807 +0000 UTC m=+1150.679532295" watchObservedRunningTime="2026-10-31 23:08:51.933783017 +0000 UTC m=+1150.680972524"
Oct 31 23:08:52.384767 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:52.384731    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ee9b5ef-cded-4849-8ebc-86c305c4e8a7-maas-api-tls\") pod \"maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg\" (UID: \"7ee9b5ef-cded-4849-8ebc-86c305c4e8a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg"
Oct 31 23:08:52.387104 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:52.387077    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ee9b5ef-cded-4849-8ebc-86c305c4e8a7-maas-api-tls\") pod \"maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg\" (UID: \"7ee9b5ef-cded-4849-8ebc-86c305c4e8a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg"
Oct 31 23:08:52.629850 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:52.629815    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg"
Oct 31 23:08:52.907165 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:52.907091    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg"]
Oct 31 23:08:52.909974 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:08:52.909948    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee9b5ef_cded_4849_8ebc_86c305c4e8a7.slice/crio-2568db3705e7da8cb0159e76ccce3ef1e3b7e29d5d6044bfc246bd4c49bf9bcf WatchSource:0}: Error finding container 2568db3705e7da8cb0159e76ccce3ef1e3b7e29d5d6044bfc246bd4c49bf9bcf: Status 404 returned error can't find the container with id 2568db3705e7da8cb0159e76ccce3ef1e3b7e29d5d6044bfc246bd4c49bf9bcf
Oct 31 23:08:53.011131 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:53.011097    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:08:53.910382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:53.910331    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg" event={"ID":"7ee9b5ef-cded-4849-8ebc-86c305c4e8a7","Type":"ContainerStarted","Data":"d36b7c285acd572606a5d13337f21d5cc35bddd95af24fc52b9474678913f385"}
Oct 31 23:08:53.910750 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:53.910402    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg" event={"ID":"7ee9b5ef-cded-4849-8ebc-86c305c4e8a7","Type":"ContainerStarted","Data":"2568db3705e7da8cb0159e76ccce3ef1e3b7e29d5d6044bfc246bd4c49bf9bcf"}
Oct 31 23:08:53.910750 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:53.910431    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg"
Oct 31 23:08:53.931120 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:53.931076    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg" podStartSLOduration=2.482126137 podStartE2EDuration="2.931061457s" totalImagesPullingTime="448.93532ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:53.929760211 +0000 UTC m=+1152.676949716" watchObservedRunningTime="2026-10-31 23:08:53.931061457 +0000 UTC m=+1152.678250974"
Oct 31 23:08:57.910964 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:57.910929    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr"
Oct 31 23:08:59.922484 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:08:59.922456    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg"
Oct 31 23:09:01.986239 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:01.986205    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:09:02.228111 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:02.228075    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"]
Oct 31 23:09:02.232240 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:02.232214    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:09:02.236177 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:02.236156    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-135248\"/\"kube-root-ca.crt\""
Oct 31 23:09:02.236329 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:02.236209    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-135248\"/\"openshift-service-ca.crt\""
Oct 31 23:09:02.236909 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:02.236888    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-135248\"/\"default-dockercfg-s6wqs\""
Oct 31 23:09:02.237008 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:02.236929    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-135248\"/\"simulator-w4-135248-kserve-self-signed-certs\""
Oct 31 23:09:02.241242 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:02.241218    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"]
Oct 31 23:09:02.281561 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:02.281525    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd9c-f413-4852-869c-adc53259a977-dshm\") pod \"simulator-w4-135248-kserve-6b9dcfcd66-t29w9\" (UID: \"fa98bd9c-f413-4852-869c-adc53259a977\") " pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:09:02.281754 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:02.281567    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd9c-f413-4852-869c-adc53259a977-model-cache\") pod \"simulator-w4-135248-kserve-6b9dcfcd66-t29w9\" (UID: \"fa98bd9c-f413-4852-869c-adc53259a977\") " pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:09:02.281754 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:02.281644    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa98bd9c-f413-4852-869c-adc53259a977-tls-certs\") pod \"simulator-w4-135248-kserve-6b9dcfcd66-t29w9\" (UID: \"fa98bd9c-f413-4852-869c-adc53259a977\") " pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:09:02.281754 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:02.281667    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rxtm\" (UniqueName: \"kubernetes.io/projected/fa98bd9c-f413-4852-869c-adc53259a977-kube-api-access-8rxtm\") pod \"simulator-w4-135248-kserve-6b9dcfcd66-t29w9\" (UID: \"fa98bd9c-f413-4852-869c-adc53259a977\") " pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:09:02.281972 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:02.281829    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd9c-f413-4852-869c-adc53259a977-home\") pod \"simulator-w4-135248-kserve-6b9dcfcd66-t29w9\" (UID: \"fa98bd9c-f413-4852-869c-adc53259a977\") " pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:09:02.281972 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:02.281908    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd9c-f413-4852-869c-adc53259a977-tmp-dir\") pod \"simulator-w4-135248-kserve-6b9dcfcd66-t29w9\" (UID: \"fa98bd9c-f413-4852-869c-adc53259a977\") " pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:09:02.382887 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:02.382853    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd9c-f413-4852-869c-adc53259a977-home\") pod \"simulator-w4-135248-kserve-6b9dcfcd66-t29w9\" (UID: \"fa98bd9c-f413-4852-869c-adc53259a977\") " pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:09:02.383062 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:02.382899    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd9c-f413-4852-869c-adc53259a977-tmp-dir\") pod \"simulator-w4-135248-kserve-6b9dcfcd66-t29w9\" (UID: \"fa98bd9c-f413-4852-869c-adc53259a977\") " pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:09:02.383062 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:02.382925    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd9c-f413-4852-869c-adc53259a977-dshm\") pod \"simulator-w4-135248-kserve-6b9dcfcd66-t29w9\" (UID: \"fa98bd9c-f413-4852-869c-adc53259a977\") " pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:09:02.383062 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:02.382939    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd9c-f413-4852-869c-adc53259a977-model-cache\") pod \"simulator-w4-135248-kserve-6b9dcfcd66-t29w9\" (UID: \"fa98bd9c-f413-4852-869c-adc53259a977\") " pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:09:02.383062 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:02.382978    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa98bd9c-f413-4852-869c-adc53259a977-tls-certs\") pod \"simulator-w4-135248-kserve-6b9dcfcd66-t29w9\" (UID: \"fa98bd9c-f413-4852-869c-adc53259a977\") " pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:09:02.383062 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:02.382996    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rxtm\" (UniqueName: \"kubernetes.io/projected/fa98bd9c-f413-4852-869c-adc53259a977-kube-api-access-8rxtm\") pod \"simulator-w4-135248-kserve-6b9dcfcd66-t29w9\" (UID: \"fa98bd9c-f413-4852-869c-adc53259a977\") " pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:09:02.383389 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:02.383342    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd9c-f413-4852-869c-adc53259a977-tmp-dir\") pod \"simulator-w4-135248-kserve-6b9dcfcd66-t29w9\" (UID: \"fa98bd9c-f413-4852-869c-adc53259a977\") " pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:09:02.383438 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:02.383353    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd9c-f413-4852-869c-adc53259a977-home\") pod \"simulator-w4-135248-kserve-6b9dcfcd66-t29w9\" (UID: \"fa98bd9c-f413-4852-869c-adc53259a977\") " pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:09:02.383476 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:02.383457    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd9c-f413-4852-869c-adc53259a977-model-cache\") pod \"simulator-w4-135248-kserve-6b9dcfcd66-t29w9\" (UID: \"fa98bd9c-f413-4852-869c-adc53259a977\") " pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:09:02.385291 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:02.385272    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd9c-f413-4852-869c-adc53259a977-dshm\") pod \"simulator-w4-135248-kserve-6b9dcfcd66-t29w9\" (UID: \"fa98bd9c-f413-4852-869c-adc53259a977\") " pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:09:02.385611 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:02.385583    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa98bd9c-f413-4852-869c-adc53259a977-tls-certs\") pod \"simulator-w4-135248-kserve-6b9dcfcd66-t29w9\" (UID: \"fa98bd9c-f413-4852-869c-adc53259a977\") " pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:09:02.394861 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:02.394839    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rxtm\" (UniqueName: \"kubernetes.io/projected/fa98bd9c-f413-4852-869c-adc53259a977-kube-api-access-8rxtm\") pod \"simulator-w4-135248-kserve-6b9dcfcd66-t29w9\" (UID: \"fa98bd9c-f413-4852-869c-adc53259a977\") " pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:09:02.545230 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:02.545152    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:09:02.546158 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:09:02.546123    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:09:02.865440 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:02.865117    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"]
Oct 31 23:09:02.870748 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:09:02.870701    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa98bd9c_f413_4852_869c_adc53259a977.slice/crio-c54889f9fe618bc9cea456b988366c175ec6d579af11d3f96ee37efd9f81a939 WatchSource:0}: Error finding container c54889f9fe618bc9cea456b988366c175ec6d579af11d3f96ee37efd9f81a939: Status 404 returned error can't find the container with id c54889f9fe618bc9cea456b988366c175ec6d579af11d3f96ee37efd9f81a939
Oct 31 23:09:02.914433 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:02.913681    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d7eed-7c97f4ff9f-whnrt"
Oct 31 23:09:02.996396 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:02.995427    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" event={"ID":"fa98bd9c-f413-4852-869c-adc53259a977","Type":"ContainerStarted","Data":"c54889f9fe618bc9cea456b988366c175ec6d579af11d3f96ee37efd9f81a939"}
Oct 31 23:09:06.011671 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:06.011633    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" event={"ID":"fa98bd9c-f413-4852-869c-adc53259a977","Type":"ContainerStarted","Data":"aaeaff5fa1f733c19d17d6f7f7af6c9b749894159431c70fb2f4004c78cb9092"}
Oct 31 23:09:06.052872 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:06.052799    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podStartSLOduration=1.943169197 podStartE2EDuration="4.052780107s" totalImagesPullingTime="2.10961091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:09:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:09:06.042913585 +0000 UTC m=+1164.790103091" watchObservedRunningTime="2026-10-31 23:09:06.052780107 +0000 UTC m=+1164.799969615"
Oct 31 23:09:06.555005 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:06.554943    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:09:09.704402 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:09.702460    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"]
Oct 31 23:09:09.708542 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:09.708440    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:09:09.725208 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:09.725176    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-255748\"/\"kube-root-ca.crt\""
Oct 31 23:09:09.725432 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:09.725328    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-255748\"/\"default-dockercfg-5rkmz\""
Oct 31 23:09:09.725512 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:09.725455    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-255748\"/\"simulator-w2-255748-kserve-self-signed-certs\""
Oct 31 23:09:09.727005 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:09.726954    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-255748\"/\"openshift-service-ca.crt\""
Oct 31 23:09:09.727889 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:09.727857    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"]
Oct 31 23:09:09.795783 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:09.795672    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1cc0440-3a85-456d-981a-915ae2f4e0bb-model-cache\") pod \"simulator-w2-255748-kserve-dff978c5d-hglpz\" (UID: \"b1cc0440-3a85-456d-981a-915ae2f4e0bb\") " pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:09:09.795783 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:09.795713    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-748tb\" (UniqueName: \"kubernetes.io/projected/b1cc0440-3a85-456d-981a-915ae2f4e0bb-kube-api-access-748tb\") pod \"simulator-w2-255748-kserve-dff978c5d-hglpz\" (UID: \"b1cc0440-3a85-456d-981a-915ae2f4e0bb\") " pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:09:09.796022 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:09.795919    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1cc0440-3a85-456d-981a-915ae2f4e0bb-tls-certs\") pod \"simulator-w2-255748-kserve-dff978c5d-hglpz\" (UID: \"b1cc0440-3a85-456d-981a-915ae2f4e0bb\") " pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:09:09.796022 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:09.796009    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1cc0440-3a85-456d-981a-915ae2f4e0bb-tmp-dir\") pod \"simulator-w2-255748-kserve-dff978c5d-hglpz\" (UID: \"b1cc0440-3a85-456d-981a-915ae2f4e0bb\") " pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:09:09.796152 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:09.796134    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1cc0440-3a85-456d-981a-915ae2f4e0bb-dshm\") pod \"simulator-w2-255748-kserve-dff978c5d-hglpz\" (UID: \"b1cc0440-3a85-456d-981a-915ae2f4e0bb\") " pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:09:09.796224 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:09.796164    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1cc0440-3a85-456d-981a-915ae2f4e0bb-home\") pod \"simulator-w2-255748-kserve-dff978c5d-hglpz\" (UID: \"b1cc0440-3a85-456d-981a-915ae2f4e0bb\") " pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:09:09.897764 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:09.897725    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1cc0440-3a85-456d-981a-915ae2f4e0bb-dshm\") pod \"simulator-w2-255748-kserve-dff978c5d-hglpz\" (UID: \"b1cc0440-3a85-456d-981a-915ae2f4e0bb\") " pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:09:09.897764 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:09.897772    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1cc0440-3a85-456d-981a-915ae2f4e0bb-home\") pod \"simulator-w2-255748-kserve-dff978c5d-hglpz\" (UID: \"b1cc0440-3a85-456d-981a-915ae2f4e0bb\") " pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:09:09.898035 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:09.897816    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1cc0440-3a85-456d-981a-915ae2f4e0bb-model-cache\") pod \"simulator-w2-255748-kserve-dff978c5d-hglpz\" (UID: \"b1cc0440-3a85-456d-981a-915ae2f4e0bb\") " pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:09:09.898035 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:09.897840    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-748tb\" (UniqueName: \"kubernetes.io/projected/b1cc0440-3a85-456d-981a-915ae2f4e0bb-kube-api-access-748tb\") pod \"simulator-w2-255748-kserve-dff978c5d-hglpz\" (UID: \"b1cc0440-3a85-456d-981a-915ae2f4e0bb\") " pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:09:09.898035 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:09.897926    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1cc0440-3a85-456d-981a-915ae2f4e0bb-tls-certs\") pod \"simulator-w2-255748-kserve-dff978c5d-hglpz\" (UID: \"b1cc0440-3a85-456d-981a-915ae2f4e0bb\") " pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:09:09.898035 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:09.897953    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1cc0440-3a85-456d-981a-915ae2f4e0bb-tmp-dir\") pod \"simulator-w2-255748-kserve-dff978c5d-hglpz\" (UID: \"b1cc0440-3a85-456d-981a-915ae2f4e0bb\") " pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:09:09.898344 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:09.898231    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1cc0440-3a85-456d-981a-915ae2f4e0bb-home\") pod \"simulator-w2-255748-kserve-dff978c5d-hglpz\" (UID: \"b1cc0440-3a85-456d-981a-915ae2f4e0bb\") " pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:09:09.898458 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:09.898435    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1cc0440-3a85-456d-981a-915ae2f4e0bb-tmp-dir\") pod \"simulator-w2-255748-kserve-dff978c5d-hglpz\" (UID: \"b1cc0440-3a85-456d-981a-915ae2f4e0bb\") " pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:09:09.898527 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:09.898462    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1cc0440-3a85-456d-981a-915ae2f4e0bb-model-cache\") pod \"simulator-w2-255748-kserve-dff978c5d-hglpz\" (UID: \"b1cc0440-3a85-456d-981a-915ae2f4e0bb\") " pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:09:09.901017 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:09.900992    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1cc0440-3a85-456d-981a-915ae2f4e0bb-tls-certs\") pod \"simulator-w2-255748-kserve-dff978c5d-hglpz\" (UID: \"b1cc0440-3a85-456d-981a-915ae2f4e0bb\") " pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:09:09.901801 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:09.901777    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1cc0440-3a85-456d-981a-915ae2f4e0bb-dshm\") pod \"simulator-w2-255748-kserve-dff978c5d-hglpz\" (UID: \"b1cc0440-3a85-456d-981a-915ae2f4e0bb\") " pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:09:09.917245 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:09.917180    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-748tb\" (UniqueName: \"kubernetes.io/projected/b1cc0440-3a85-456d-981a-915ae2f4e0bb-kube-api-access-748tb\") pod \"simulator-w2-255748-kserve-dff978c5d-hglpz\" (UID: \"b1cc0440-3a85-456d-981a-915ae2f4e0bb\") " pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:09:10.054615 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:10.054533    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:09:10.363536 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:10.363459    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"]
Oct 31 23:09:10.367392 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:09:10.367340    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1cc0440_3a85_456d_981a_915ae2f4e0bb.slice/crio-3eefd83de4bd3586d9702b06ab663a75003300e56bc51b705b3c6735b3479a23 WatchSource:0}: Error finding container 3eefd83de4bd3586d9702b06ab663a75003300e56bc51b705b3c6735b3479a23: Status 404 returned error can't find the container with id 3eefd83de4bd3586d9702b06ab663a75003300e56bc51b705b3c6735b3479a23
Oct 31 23:09:11.036365 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:11.036310    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" event={"ID":"b1cc0440-3a85-456d-981a-915ae2f4e0bb","Type":"ContainerStarted","Data":"ad356fb1ac3c6dbac69dd222c782823e58db0471c2948d13dd7f17193a8de41b"}
Oct 31 23:09:11.036779 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:11.036409    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" event={"ID":"b1cc0440-3a85-456d-981a-915ae2f4e0bb","Type":"ContainerStarted","Data":"3eefd83de4bd3586d9702b06ab663a75003300e56bc51b705b3c6735b3479a23"}
Oct 31 23:09:11.055257 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:11.055199    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podStartSLOduration=2.055180861 podStartE2EDuration="2.055180861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:09:11.054236189 +0000 UTC m=+1169.801425694" watchObservedRunningTime="2026-10-31 23:09:11.055180861 +0000 UTC m=+1169.802370367"
Oct 31 23:09:11.055257 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:11.055243    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:09:11.403469 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:11.403439    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:09:11.843724 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:11.843631    4301 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 31 23:09:11.843724 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:11.843685    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg" podUID="d0195c12-01bd-4f4a-a72a-b608150516f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 31 23:09:12.043220 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:12.043189    4301 generic.go:356] "Generic (PLEG): container finished" podID="d0195c12-01bd-4f4a-a72a-b608150516f2" containerID="ef96b32c0a70a58bf905ba846040660f49d77848cc1a7acb6c6e6567c960e7b5" exitCode=0
Oct 31 23:09:12.043634 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:12.043266    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg" event={"ID":"d0195c12-01bd-4f4a-a72a-b608150516f2","Type":"ContainerDied","Data":"ef96b32c0a70a58bf905ba846040660f49d77848cc1a7acb6c6e6567c960e7b5"}
Oct 31 23:09:12.043759 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:12.043745    4301 scope.go:122] "RemoveContainer" containerID="ef96b32c0a70a58bf905ba846040660f49d77848cc1a7acb6c6e6567c960e7b5"
Oct 31 23:09:12.044947 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:12.044919    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:09:12.045038 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:12.044985    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:09:12.045038 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:12.045025    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:09:12.546382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:12.545799    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:09:12.566769 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:12.566742    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:09:13.059381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:13.058269    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg" event={"ID":"d0195c12-01bd-4f4a-a72a-b608150516f2","Type":"ContainerStarted","Data":"b55f6e37daaf160cc50371071ce1abe701dcc03711497ddb0828c5bcba7adb06"}
Oct 31 23:09:13.070457 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:13.070433    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:09:13.760542 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:13.760495    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:09:13.760831 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:13.760797    4301 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 31 23:09:13.760911 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:13.760848    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg" podUID="d0195c12-01bd-4f4a-a72a-b608150516f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 31 23:09:14.760422 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:14.760332    4301 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 31 23:09:14.760929 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:14.760436    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg" podUID="d0195c12-01bd-4f4a-a72a-b608150516f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 31 23:09:15.760696 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:15.760663    4301 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 31 23:09:15.761172 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:15.760730    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg" podUID="d0195c12-01bd-4f4a-a72a-b608150516f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 31 23:09:16.226298 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:16.226260    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"]
Oct 31 23:09:16.249558 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:16.249528    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:09:16.264379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:16.257514    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"]
Oct 31 23:09:16.264379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:16.258343    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-67fce9\"/\"openshift-service-ca.crt\""
Oct 31 23:09:16.264379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:16.263555    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-67fce9\"/\"kube-root-ca.crt\""
Oct 31 23:09:16.264379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:16.263587    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-67fce9\"/\"default-dockercfg-7cn9l\""
Oct 31 23:09:16.264379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:16.263633    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-67fce9\"/\"simulator-w3-67fce9-kserve-self-signed-certs\""
Oct 31 23:09:16.293107 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:16.293060    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1486aeca-b15c-4611-beda-0baf2e503a89-model-cache\") pod \"simulator-w3-67fce9-kserve-9c6ddd964-46q2t\" (UID: \"1486aeca-b15c-4611-beda-0baf2e503a89\") " pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:09:16.293283 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:16.293166    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1486aeca-b15c-4611-beda-0baf2e503a89-tls-certs\") pod \"simulator-w3-67fce9-kserve-9c6ddd964-46q2t\" (UID: \"1486aeca-b15c-4611-beda-0baf2e503a89\") " pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:09:16.293283 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:16.293218    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz2lh\" (UniqueName: \"kubernetes.io/projected/1486aeca-b15c-4611-beda-0baf2e503a89-kube-api-access-vz2lh\") pod \"simulator-w3-67fce9-kserve-9c6ddd964-46q2t\" (UID: \"1486aeca-b15c-4611-beda-0baf2e503a89\") " pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:09:16.293283 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:16.293248    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1486aeca-b15c-4611-beda-0baf2e503a89-home\") pod \"simulator-w3-67fce9-kserve-9c6ddd964-46q2t\" (UID: \"1486aeca-b15c-4611-beda-0baf2e503a89\") " pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:09:16.293283 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:16.293269    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1486aeca-b15c-4611-beda-0baf2e503a89-dshm\") pod \"simulator-w3-67fce9-kserve-9c6ddd964-46q2t\" (UID: \"1486aeca-b15c-4611-beda-0baf2e503a89\") " pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:09:16.293561 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:16.293446    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1486aeca-b15c-4611-beda-0baf2e503a89-tmp-dir\") pod \"simulator-w3-67fce9-kserve-9c6ddd964-46q2t\" (UID: \"1486aeca-b15c-4611-beda-0baf2e503a89\") " pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:09:16.397070 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:16.397027    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz2lh\" (UniqueName: \"kubernetes.io/projected/1486aeca-b15c-4611-beda-0baf2e503a89-kube-api-access-vz2lh\") pod \"simulator-w3-67fce9-kserve-9c6ddd964-46q2t\" (UID: \"1486aeca-b15c-4611-beda-0baf2e503a89\") " pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:09:16.397326 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:16.397303    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1486aeca-b15c-4611-beda-0baf2e503a89-home\") pod \"simulator-w3-67fce9-kserve-9c6ddd964-46q2t\" (UID: \"1486aeca-b15c-4611-beda-0baf2e503a89\") " pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:09:16.397476 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:16.397460    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1486aeca-b15c-4611-beda-0baf2e503a89-dshm\") pod \"simulator-w3-67fce9-kserve-9c6ddd964-46q2t\" (UID: \"1486aeca-b15c-4611-beda-0baf2e503a89\") " pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:09:16.397631 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:16.397616    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1486aeca-b15c-4611-beda-0baf2e503a89-tmp-dir\") pod \"simulator-w3-67fce9-kserve-9c6ddd964-46q2t\" (UID: \"1486aeca-b15c-4611-beda-0baf2e503a89\") " pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:09:16.397820 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:16.397805    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1486aeca-b15c-4611-beda-0baf2e503a89-model-cache\") pod \"simulator-w3-67fce9-kserve-9c6ddd964-46q2t\" (UID: \"1486aeca-b15c-4611-beda-0baf2e503a89\") " pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:09:16.397946 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:16.397930    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1486aeca-b15c-4611-beda-0baf2e503a89-tls-certs\") pod \"simulator-w3-67fce9-kserve-9c6ddd964-46q2t\" (UID: \"1486aeca-b15c-4611-beda-0baf2e503a89\") " pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:09:16.399779 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:16.399166    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1486aeca-b15c-4611-beda-0baf2e503a89-model-cache\") pod \"simulator-w3-67fce9-kserve-9c6ddd964-46q2t\" (UID: \"1486aeca-b15c-4611-beda-0baf2e503a89\") " pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:09:16.399779 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:16.399477    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1486aeca-b15c-4611-beda-0baf2e503a89-tmp-dir\") pod \"simulator-w3-67fce9-kserve-9c6ddd964-46q2t\" (UID: \"1486aeca-b15c-4611-beda-0baf2e503a89\") " pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:09:16.399779 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:16.399733    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1486aeca-b15c-4611-beda-0baf2e503a89-home\") pod \"simulator-w3-67fce9-kserve-9c6ddd964-46q2t\" (UID: \"1486aeca-b15c-4611-beda-0baf2e503a89\") " pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:09:16.419997 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:16.419928    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz2lh\" (UniqueName: \"kubernetes.io/projected/1486aeca-b15c-4611-beda-0baf2e503a89-kube-api-access-vz2lh\") pod \"simulator-w3-67fce9-kserve-9c6ddd964-46q2t\" (UID: \"1486aeca-b15c-4611-beda-0baf2e503a89\") " pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:09:16.420240 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:16.420194    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1486aeca-b15c-4611-beda-0baf2e503a89-tls-certs\") pod \"simulator-w3-67fce9-kserve-9c6ddd964-46q2t\" (UID: \"1486aeca-b15c-4611-beda-0baf2e503a89\") " pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:09:16.424321 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:16.424296    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1486aeca-b15c-4611-beda-0baf2e503a89-dshm\") pod \"simulator-w3-67fce9-kserve-9c6ddd964-46q2t\" (UID: \"1486aeca-b15c-4611-beda-0baf2e503a89\") " pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:09:16.581882 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:16.581814    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:09:16.760619 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:16.760577    4301 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 31 23:09:16.760813 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:16.760653    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg" podUID="d0195c12-01bd-4f4a-a72a-b608150516f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 31 23:09:16.879749 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:16.879711    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"]
Oct 31 23:09:16.882865 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:09:16.882841    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1486aeca_b15c_4611_beda_0baf2e503a89.slice/crio-c779680878376688d2c41159305275e30013e2f1cdfc84f589f90814191df7f3 WatchSource:0}: Error finding container c779680878376688d2c41159305275e30013e2f1cdfc84f589f90814191df7f3: Status 404 returned error can't find the container with id c779680878376688d2c41159305275e30013e2f1cdfc84f589f90814191df7f3
Oct 31 23:09:17.078958 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:17.078927    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" event={"ID":"1486aeca-b15c-4611-beda-0baf2e503a89","Type":"ContainerStarted","Data":"d13f078c8c0a8d358fc427ef0605270c01dc7a5937327089a1f781d81d4645b5"}
Oct 31 23:09:17.078958 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:17.078966    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" event={"ID":"1486aeca-b15c-4611-beda-0baf2e503a89","Type":"ContainerStarted","Data":"c779680878376688d2c41159305275e30013e2f1cdfc84f589f90814191df7f3"}
Oct 31 23:09:17.102378 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:17.102329    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podStartSLOduration=1.102315528 podStartE2EDuration="1.102315528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:09:17.101943214 +0000 UTC m=+1175.849132714" watchObservedRunningTime="2026-10-31 23:09:17.102315528 +0000 UTC m=+1175.849505032"
Oct 31 23:09:17.582250 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:17.582217    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:09:17.761160 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:17.761120    4301 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 31 23:09:17.761690 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:17.761171    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg" podUID="d0195c12-01bd-4f4a-a72a-b608150516f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 31 23:09:18.765045 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:18.765019    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:09:18.765582 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:18.765394    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:09:18.766175 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:18.766156    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:09:20.055202 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:20.055166    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:09:20.071919 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:20.071888    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:09:20.105658 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:20.105628    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:09:26.582146 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:26.582109    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:09:26.594148 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:26.594125    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:09:26.692122 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:26.692080    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"]
Oct 31 23:09:26.698491 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:26.698462    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:09:26.708641 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:26.708612    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-135248\"/\"premium-w4-135248-kserve-self-signed-certs\""
Oct 31 23:09:26.710244 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:26.710214    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"]
Oct 31 23:09:26.803784 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:26.803747    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-tmp-dir\") pod \"premium-w4-135248-kserve-6474f475c5-lxh6m\" (UID: \"289e47fa-d3c8-49b4-aac7-0cd0959e64aa\") " pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:09:26.803985 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:26.803802    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-home\") pod \"premium-w4-135248-kserve-6474f475c5-lxh6m\" (UID: \"289e47fa-d3c8-49b4-aac7-0cd0959e64aa\") " pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:09:26.803985 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:26.803957    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx8t5\" (UniqueName: \"kubernetes.io/projected/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-kube-api-access-mx8t5\") pod \"premium-w4-135248-kserve-6474f475c5-lxh6m\" (UID: \"289e47fa-d3c8-49b4-aac7-0cd0959e64aa\") " pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:09:26.804075 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:26.804030    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-dshm\") pod \"premium-w4-135248-kserve-6474f475c5-lxh6m\" (UID: \"289e47fa-d3c8-49b4-aac7-0cd0959e64aa\") " pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:09:26.804156 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:26.804137    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-tls-certs\") pod \"premium-w4-135248-kserve-6474f475c5-lxh6m\" (UID: \"289e47fa-d3c8-49b4-aac7-0cd0959e64aa\") " pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:09:26.804285 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:26.804262    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-model-cache\") pod \"premium-w4-135248-kserve-6474f475c5-lxh6m\" (UID: \"289e47fa-d3c8-49b4-aac7-0cd0959e64aa\") " pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:09:26.904937 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:26.904901    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx8t5\" (UniqueName: \"kubernetes.io/projected/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-kube-api-access-mx8t5\") pod \"premium-w4-135248-kserve-6474f475c5-lxh6m\" (UID: \"289e47fa-d3c8-49b4-aac7-0cd0959e64aa\") " pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:09:26.904937 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:26.904939    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-dshm\") pod \"premium-w4-135248-kserve-6474f475c5-lxh6m\" (UID: \"289e47fa-d3c8-49b4-aac7-0cd0959e64aa\") " pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:09:26.905185 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:26.904998    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-tls-certs\") pod \"premium-w4-135248-kserve-6474f475c5-lxh6m\" (UID: \"289e47fa-d3c8-49b4-aac7-0cd0959e64aa\") " pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:09:26.905185 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:26.905021    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-model-cache\") pod \"premium-w4-135248-kserve-6474f475c5-lxh6m\" (UID: \"289e47fa-d3c8-49b4-aac7-0cd0959e64aa\") " pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:09:26.905185 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:26.905045    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-tmp-dir\") pod \"premium-w4-135248-kserve-6474f475c5-lxh6m\" (UID: \"289e47fa-d3c8-49b4-aac7-0cd0959e64aa\") " pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:09:26.905185 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:26.905068    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-home\") pod \"premium-w4-135248-kserve-6474f475c5-lxh6m\" (UID: \"289e47fa-d3c8-49b4-aac7-0cd0959e64aa\") " pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:09:26.905536 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:26.905509    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-model-cache\") pod \"premium-w4-135248-kserve-6474f475c5-lxh6m\" (UID: \"289e47fa-d3c8-49b4-aac7-0cd0959e64aa\") " pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:09:26.905653 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:26.905570    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-home\") pod \"premium-w4-135248-kserve-6474f475c5-lxh6m\" (UID: \"289e47fa-d3c8-49b4-aac7-0cd0959e64aa\") " pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:09:26.905653 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:26.905610    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-tmp-dir\") pod \"premium-w4-135248-kserve-6474f475c5-lxh6m\" (UID: \"289e47fa-d3c8-49b4-aac7-0cd0959e64aa\") " pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:09:26.907504 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:26.907486    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-dshm\") pod \"premium-w4-135248-kserve-6474f475c5-lxh6m\" (UID: \"289e47fa-d3c8-49b4-aac7-0cd0959e64aa\") " pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:09:26.907703 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:26.907666    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-tls-certs\") pod \"premium-w4-135248-kserve-6474f475c5-lxh6m\" (UID: \"289e47fa-d3c8-49b4-aac7-0cd0959e64aa\") " pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:09:26.916422 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:26.916400    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx8t5\" (UniqueName: \"kubernetes.io/projected/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-kube-api-access-mx8t5\") pod \"premium-w4-135248-kserve-6474f475c5-lxh6m\" (UID: \"289e47fa-d3c8-49b4-aac7-0cd0959e64aa\") " pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:09:27.013225 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:27.013188    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:09:27.145206 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:27.145177    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:09:27.294778 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:27.294687    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"]
Oct 31 23:09:27.297575 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:09:27.297546    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod289e47fa_d3c8_49b4_aac7_0cd0959e64aa.slice/crio-27f5360a66a09b0abf979fbb6dacc0e3ca93f7911104fa151769e9877bb3175a WatchSource:0}: Error finding container 27f5360a66a09b0abf979fbb6dacc0e3ca93f7911104fa151769e9877bb3175a: Status 404 returned error can't find the container with id 27f5360a66a09b0abf979fbb6dacc0e3ca93f7911104fa151769e9877bb3175a
Oct 31 23:09:28.141219 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:28.141177    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" event={"ID":"289e47fa-d3c8-49b4-aac7-0cd0959e64aa","Type":"ContainerStarted","Data":"cd1a83d5f8ec000fbccaaa6a674ea6b1967c3ad6015a9c1c35dbd6f9b1dc9b55"}
Oct 31 23:09:28.141705 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:28.141237    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" event={"ID":"289e47fa-d3c8-49b4-aac7-0cd0959e64aa","Type":"ContainerStarted","Data":"27f5360a66a09b0abf979fbb6dacc0e3ca93f7911104fa151769e9877bb3175a"}
Oct 31 23:09:28.160817 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:28.160773    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podStartSLOduration=2.160760201 podStartE2EDuration="2.160760201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:09:28.159329796 +0000 UTC m=+1186.906519299" watchObservedRunningTime="2026-10-31 23:09:28.160760201 +0000 UTC m=+1186.907949717"
Oct 31 23:09:28.339594 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:28.339563    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:09:29.016480 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:29.014750    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:09:32.545014 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:09:32.544975    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:09:37.013763 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:37.013708    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:09:37.026347 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:37.026324    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:09:37.194165 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:37.194135    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:09:41.486898 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:41.486865    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s9nl_1f0042ee-6258-4aa8-af48-7568f2f2e8a1/ovn-acl-logging/0.log"
Oct 31 23:09:41.491753 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:41.491728    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s9nl_1f0042ee-6258-4aa8-af48-7568f2f2e8a1/ovn-acl-logging/0.log"
Oct 31 23:09:42.535591 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:42.535558    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"]
Oct 31 23:09:42.539003 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:42.538985    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:09:42.545567 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:42.545542    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a16dfb\"/\"kube-root-ca.crt\""
Oct 31 23:09:42.548182 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:42.548164    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a16dfb\"/\"openshift-service-ca.crt\""
Oct 31 23:09:42.549117 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:42.549097    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a16dfb\"/\"default-dockercfg-wzjmz\""
Oct 31 23:09:42.551436 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:42.549453    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a16dfb\"/\"premium-w0-a16dfb-kserve-self-signed-certs\""
Oct 31 23:09:42.551687 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:42.551662    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"]
Oct 31 23:09:42.699381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:42.690114    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff4f6df9-e1f0-4163-895c-7df6b065b033-home\") pod \"premium-w0-a16dfb-kserve-64c9b45554-bckr4\" (UID: \"ff4f6df9-e1f0-4163-895c-7df6b065b033\") " pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:09:42.699381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:42.690168    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff4f6df9-e1f0-4163-895c-7df6b065b033-model-cache\") pod \"premium-w0-a16dfb-kserve-64c9b45554-bckr4\" (UID: \"ff4f6df9-e1f0-4163-895c-7df6b065b033\") " pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:09:42.699381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:42.690228    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff4f6df9-e1f0-4163-895c-7df6b065b033-dshm\") pod \"premium-w0-a16dfb-kserve-64c9b45554-bckr4\" (UID: \"ff4f6df9-e1f0-4163-895c-7df6b065b033\") " pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:09:42.699381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:42.690264    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4qgw\" (UniqueName: \"kubernetes.io/projected/ff4f6df9-e1f0-4163-895c-7df6b065b033-kube-api-access-j4qgw\") pod \"premium-w0-a16dfb-kserve-64c9b45554-bckr4\" (UID: \"ff4f6df9-e1f0-4163-895c-7df6b065b033\") " pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:09:42.699381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:42.690372    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff4f6df9-e1f0-4163-895c-7df6b065b033-tmp-dir\") pod \"premium-w0-a16dfb-kserve-64c9b45554-bckr4\" (UID: \"ff4f6df9-e1f0-4163-895c-7df6b065b033\") " pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:09:42.699381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:42.690451    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff4f6df9-e1f0-4163-895c-7df6b065b033-tls-certs\") pod \"premium-w0-a16dfb-kserve-64c9b45554-bckr4\" (UID: \"ff4f6df9-e1f0-4163-895c-7df6b065b033\") " pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:09:42.791235 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:42.791143    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff4f6df9-e1f0-4163-895c-7df6b065b033-home\") pod \"premium-w0-a16dfb-kserve-64c9b45554-bckr4\" (UID: \"ff4f6df9-e1f0-4163-895c-7df6b065b033\") " pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:09:42.791235 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:42.791184    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff4f6df9-e1f0-4163-895c-7df6b065b033-model-cache\") pod \"premium-w0-a16dfb-kserve-64c9b45554-bckr4\" (UID: \"ff4f6df9-e1f0-4163-895c-7df6b065b033\") " pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:09:42.791235 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:42.791211    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff4f6df9-e1f0-4163-895c-7df6b065b033-dshm\") pod \"premium-w0-a16dfb-kserve-64c9b45554-bckr4\" (UID: \"ff4f6df9-e1f0-4163-895c-7df6b065b033\") " pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:09:42.791235 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:42.791228    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4qgw\" (UniqueName: \"kubernetes.io/projected/ff4f6df9-e1f0-4163-895c-7df6b065b033-kube-api-access-j4qgw\") pod \"premium-w0-a16dfb-kserve-64c9b45554-bckr4\" (UID: \"ff4f6df9-e1f0-4163-895c-7df6b065b033\") " pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:09:42.791589 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:42.791274    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff4f6df9-e1f0-4163-895c-7df6b065b033-tmp-dir\") pod \"premium-w0-a16dfb-kserve-64c9b45554-bckr4\" (UID: \"ff4f6df9-e1f0-4163-895c-7df6b065b033\") " pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:09:42.791589 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:42.791310    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff4f6df9-e1f0-4163-895c-7df6b065b033-tls-certs\") pod \"premium-w0-a16dfb-kserve-64c9b45554-bckr4\" (UID: \"ff4f6df9-e1f0-4163-895c-7df6b065b033\") " pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:09:42.791715 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:42.791642    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff4f6df9-e1f0-4163-895c-7df6b065b033-model-cache\") pod \"premium-w0-a16dfb-kserve-64c9b45554-bckr4\" (UID: \"ff4f6df9-e1f0-4163-895c-7df6b065b033\") " pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:09:42.791770 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:42.791739    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff4f6df9-e1f0-4163-895c-7df6b065b033-home\") pod \"premium-w0-a16dfb-kserve-64c9b45554-bckr4\" (UID: \"ff4f6df9-e1f0-4163-895c-7df6b065b033\") " pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:09:42.791832 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:42.791814    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff4f6df9-e1f0-4163-895c-7df6b065b033-tmp-dir\") pod \"premium-w0-a16dfb-kserve-64c9b45554-bckr4\" (UID: \"ff4f6df9-e1f0-4163-895c-7df6b065b033\") " pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:09:42.793739 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:42.793714    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff4f6df9-e1f0-4163-895c-7df6b065b033-dshm\") pod \"premium-w0-a16dfb-kserve-64c9b45554-bckr4\" (UID: \"ff4f6df9-e1f0-4163-895c-7df6b065b033\") " pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:09:42.793834 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:42.793778    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff4f6df9-e1f0-4163-895c-7df6b065b033-tls-certs\") pod \"premium-w0-a16dfb-kserve-64c9b45554-bckr4\" (UID: \"ff4f6df9-e1f0-4163-895c-7df6b065b033\") " pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:09:42.806106 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:42.806074    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4qgw\" (UniqueName: \"kubernetes.io/projected/ff4f6df9-e1f0-4163-895c-7df6b065b033-kube-api-access-j4qgw\") pod \"premium-w0-a16dfb-kserve-64c9b45554-bckr4\" (UID: \"ff4f6df9-e1f0-4163-895c-7df6b065b033\") " pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:09:42.855408 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:42.855379    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:09:43.053471 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:43.050491    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:09:43.134674 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:43.134635    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"]
Oct 31 23:09:43.137833 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:09:43.137800    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff4f6df9_e1f0_4163_895c_7df6b065b033.slice/crio-d3ecfce430072ffa0e846071a01be8c2bcb2df1b00a1e062c981ed1fc228af57 WatchSource:0}: Error finding container d3ecfce430072ffa0e846071a01be8c2bcb2df1b00a1e062c981ed1fc228af57: Status 404 returned error can't find the container with id d3ecfce430072ffa0e846071a01be8c2bcb2df1b00a1e062c981ed1fc228af57
Oct 31 23:09:43.218474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:43.218436    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" event={"ID":"ff4f6df9-e1f0-4163-895c-7df6b065b033","Type":"ContainerStarted","Data":"91e981e471f18cd72cb7a666fce8f21853dbfc76270ed7ad1638c86466cb815f"}
Oct 31 23:09:43.218622 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:43.218486    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" event={"ID":"ff4f6df9-e1f0-4163-895c-7df6b065b033","Type":"ContainerStarted","Data":"d3ecfce430072ffa0e846071a01be8c2bcb2df1b00a1e062c981ed1fc228af57"}
Oct 31 23:09:43.237197 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:43.237147    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podStartSLOduration=1.237131469 podStartE2EDuration="1.237131469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:09:43.235692895 +0000 UTC m=+1201.982882401" watchObservedRunningTime="2026-10-31 23:09:43.237131469 +0000 UTC m=+1201.984321018"
Oct 31 23:09:43.856215 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:43.856176    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:09:52.855619 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:52.855583    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:09:52.868135 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:52.868105    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:09:53.278299 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:53.278267    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:09:54.245031 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.234248    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"]
Oct 31 23:09:54.251436 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.251408    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:09:54.259575 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.259549    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-2294a7-openshift-default-dockercfg-nh2x9\""
Oct 31 23:09:54.270751 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.270054    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"]
Oct 31 23:09:54.294841 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.294812    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"]
Oct 31 23:09:54.299711 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.299687    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:09:54.324509 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.322340    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"]
Oct 31 23:09:54.380318 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.380286    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-255748-openshift-default-dockercfg-89xj2\""
Oct 31 23:09:54.409926 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.409894    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istio-podinfo\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:09:54.409926 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.409929    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdxtt\" (UniqueName: \"kubernetes.io/projected/cdcadd41-cef3-4fbc-9551-ed8dae06f790-kube-api-access-pdxtt\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:09:54.410149 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.410016    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f421d513-57d6-43b1-af00-bed486a4d0bf-istio-token\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:09:54.410149 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.410044    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-credential-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:09:54.410149 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.410098    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istio-token\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:09:54.410149 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.410121    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istio-data\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:09:54.410149 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.410141    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-credential-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:09:54.410428 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.410163    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f421d513-57d6-43b1-af00-bed486a4d0bf-istiod-ca-cert\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:09:54.410428 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.410190    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-workload-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:09:54.410428 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.410262    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f421d513-57d6-43b1-af00-bed486a4d0bf-istio-podinfo\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:09:54.410575 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.410446    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-istio-envoy\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:09:54.410575 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.410482    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-istio-data\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:09:54.410575 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.410562    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-workload-certs\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:09:54.410713 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.410583    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-workload-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:09:54.410713 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.410613    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wscr6\" (UniqueName: \"kubernetes.io/projected/f421d513-57d6-43b1-af00-bed486a4d0bf-kube-api-access-wscr6\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:09:54.410713 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.410652    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istio-envoy\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:09:54.410852 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.410716    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istiod-ca-cert\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:09:54.410852 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.410759    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-workload-certs\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:09:54.491182 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.491142    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"]
Oct 31 23:09:54.495029 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.495008    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:09:54.507036 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.507004    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-67fce9-openshift-default-dockercfg-vsgs4\""
Oct 31 23:09:54.511380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.511342    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f421d513-57d6-43b1-af00-bed486a4d0bf-istio-token\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:09:54.511492 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.511479    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-credential-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:09:54.511532 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.511521    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istio-token\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:09:54.511566 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.511544    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istio-data\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:09:54.511599 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.511579    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-credential-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:09:54.511635 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.511623    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f421d513-57d6-43b1-af00-bed486a4d0bf-istiod-ca-cert\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:09:54.511689 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.511674    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-workload-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:09:54.511758 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.511719    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f421d513-57d6-43b1-af00-bed486a4d0bf-istio-podinfo\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:09:54.511812 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.511762    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-istio-envoy\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:09:54.511812 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.511788    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-istio-data\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:09:54.511911 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.511843    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-workload-certs\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:09:54.511911 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.511873    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-workload-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:09:54.511911 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.511905    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-credential-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:09:54.511998 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.511920    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wscr6\" (UniqueName: \"kubernetes.io/projected/f421d513-57d6-43b1-af00-bed486a4d0bf-kube-api-access-wscr6\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:09:54.511998 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.511934    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-credential-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:09:54.511998 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.511947    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istio-envoy\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:09:54.512088 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.511996    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istiod-ca-cert\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:09:54.512088 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.512031    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-workload-certs\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:09:54.512088 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.512073    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istio-podinfo\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:09:54.512200 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.512098    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdxtt\" (UniqueName: \"kubernetes.io/projected/cdcadd41-cef3-4fbc-9551-ed8dae06f790-kube-api-access-pdxtt\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:09:54.512200 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.511993    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-workload-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:09:54.512581 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.512559    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f421d513-57d6-43b1-af00-bed486a4d0bf-istiod-ca-cert\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:09:54.512687 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.512648    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-istio-data\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:09:54.512902 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.512873    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-workload-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:09:54.512965 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.512893    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istio-data\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:09:54.513144 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.513122    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-workload-certs\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:09:54.513407 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.513384    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-workload-certs\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:09:54.513523 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.513463    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istiod-ca-cert\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:09:54.515006 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.514629    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-istio-envoy\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:09:54.515006 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.514928    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istio-podinfo\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:09:54.515155 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.515058    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f421d513-57d6-43b1-af00-bed486a4d0bf-istio-podinfo\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:09:54.515415 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.515389    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istio-envoy\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:09:54.520171 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.520145    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"]
Oct 31 23:09:54.565381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.542106    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"]
Oct 31 23:09:54.565381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.542859    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istio-token\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:09:54.565381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.543504    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdxtt\" (UniqueName: \"kubernetes.io/projected/cdcadd41-cef3-4fbc-9551-ed8dae06f790-kube-api-access-pdxtt\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:09:54.565381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.547923    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:09:54.573656 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.567982    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f421d513-57d6-43b1-af00-bed486a4d0bf-istio-token\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:09:54.583032 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.580803    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wscr6\" (UniqueName: \"kubernetes.io/projected/f421d513-57d6-43b1-af00-bed486a4d0bf-kube-api-access-wscr6\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:09:54.611182 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.607751    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-135248-openshift-default-dockercfg-g78k6\""
Oct 31 23:09:54.611182 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.610280    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"]
Oct 31 23:09:54.612763 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.612729    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d361d757-5e4e-43a4-acb7-9ef5f24113af-istio-token\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:09:54.612888 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.612782    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-istio-envoy\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:09:54.612888 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.612832    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-workload-certs\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:09:54.613008 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.612896    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d361d757-5e4e-43a4-acb7-9ef5f24113af-istio-podinfo\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:09:54.613008 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.612926    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-istio-data\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:09:54.613008 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.612985    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28cfg\" (UniqueName: \"kubernetes.io/projected/d361d757-5e4e-43a4-acb7-9ef5f24113af-kube-api-access-28cfg\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:09:54.613168 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.613017    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-workload-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:09:54.613168 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.613039    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d361d757-5e4e-43a4-acb7-9ef5f24113af-istiod-ca-cert\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:09:54.613168 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.613076    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-credential-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:09:54.625236 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.625111    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:09:54.627029 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.626765    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:09:54.718472 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.718405    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-credential-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:09:54.726209 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.724414    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-workload-certs\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:09:54.726209 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.724482    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d361d757-5e4e-43a4-acb7-9ef5f24113af-istio-token\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:09:54.726209 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.724518    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-istio-envoy\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:09:54.726209 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.724555    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istio-envoy\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:09:54.726209 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.724632    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-workload-certs\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:09:54.726209 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.724668    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istio-data\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:09:54.726209 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.724741    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d361d757-5e4e-43a4-acb7-9ef5f24113af-istio-podinfo\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:09:54.726209 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.724772    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-istio-data\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:09:54.726209 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.724824    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-workload-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:09:54.726209 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.724869    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istio-podinfo\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:09:54.726209 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.724903    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28cfg\" (UniqueName: \"kubernetes.io/projected/d361d757-5e4e-43a4-acb7-9ef5f24113af-kube-api-access-28cfg\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:09:54.726209 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.724930    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istio-token\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:09:54.726209 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.724966    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-workload-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:09:54.726209 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.724991    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d361d757-5e4e-43a4-acb7-9ef5f24113af-istiod-ca-cert\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:09:54.726209 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.725028    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istiod-ca-cert\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:09:54.726209 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.725054    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmjhs\" (UniqueName: \"kubernetes.io/projected/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-kube-api-access-kmjhs\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:09:54.726209 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.725082    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-credential-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:09:54.726209 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.725555    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-credential-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:09:54.727202 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.726559    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-workload-certs\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:09:54.728938 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.727729    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-workload-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:09:54.728938 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.728682    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d361d757-5e4e-43a4-acb7-9ef5f24113af-istiod-ca-cert\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:09:54.728938 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.728900    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-istio-data\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:09:54.729904 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.729860    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d361d757-5e4e-43a4-acb7-9ef5f24113af-istio-podinfo\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:09:54.741613 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.741562    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d361d757-5e4e-43a4-acb7-9ef5f24113af-istio-token\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:09:54.742830 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.742760    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28cfg\" (UniqueName: \"kubernetes.io/projected/d361d757-5e4e-43a4-acb7-9ef5f24113af-kube-api-access-28cfg\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:09:54.743048 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.742989    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-istio-envoy\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:09:54.808993 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.808926    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:09:54.825795 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.825771    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-workload-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:09:54.825892 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.825813    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istio-podinfo\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:09:54.825892 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.825838    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istio-token\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:09:54.825892 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.825866    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istiod-ca-cert\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:09:54.825892 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.825889    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmjhs\" (UniqueName: \"kubernetes.io/projected/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-kube-api-access-kmjhs\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:09:54.826101 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.825936    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-credential-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:09:54.826101 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.825970    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-workload-certs\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:09:54.826101 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.826022    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istio-envoy\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:09:54.826101 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.826065    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istio-data\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:09:54.826303 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.826212    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-workload-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:09:54.826721 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.826438    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istio-data\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:09:54.826721 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.826460    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-credential-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:09:54.826721 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.826551    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-workload-certs\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:09:54.827031 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.827012    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istiod-ca-cert\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:09:54.828216 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.828191    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istio-envoy\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:09:54.828448 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.828425    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istio-podinfo\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:09:54.837236 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.837213    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istio-token\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:09:54.837497 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.837479    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmjhs\" (UniqueName: \"kubernetes.io/projected/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-kube-api-access-kmjhs\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:09:54.934588 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.934555    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:09:54.973711 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.973683    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"]
Oct 31 23:09:54.975513 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:09:54.975483    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdcadd41_cef3_4fbc_9551_ed8dae06f790.slice/crio-152b69f50aad27a1f7c30d13631cff355b04e1f958da09ffdda0e9bec758bde7 WatchSource:0}: Error finding container 152b69f50aad27a1f7c30d13631cff355b04e1f958da09ffdda0e9bec758bde7: Status 404 returned error can't find the container with id 152b69f50aad27a1f7c30d13631cff355b04e1f958da09ffdda0e9bec758bde7
Oct 31 23:09:54.976764 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.976726    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"]
Oct 31 23:09:54.978709 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.978196    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:09:54.978709 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:09:54.978231    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf421d513_57d6_43b1_af00_bed486a4d0bf.slice/crio-c66e6e07094b11a4e0cafe82215a981a290427f1d7e057e15fe4a9850c380f30 WatchSource:0}: Error finding container c66e6e07094b11a4e0cafe82215a981a290427f1d7e057e15fe4a9850c380f30: Status 404 returned error can't find the container with id c66e6e07094b11a4e0cafe82215a981a290427f1d7e057e15fe4a9850c380f30
Oct 31 23:09:54.978709 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.978285    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:09:54.978709 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.978330    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:09:54.981122 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.981088    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:09:54.981186 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.981170    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:09:54.981239 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:54.981223    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:09:55.109935 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:55.109887    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:09:55.110067 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:55.110027    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:09:55.110177 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:55.110113    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:09:55.119578 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:55.116841    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"]
Oct 31 23:09:55.234808 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:09:55.234756    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3fa555_8bcd_49d7_ac81_18cd8ddf8ab5.slice/crio-a7142c6baaaef9f5045860bc1fad336509b3c333ff1d32d0ee533ce079af8af5 WatchSource:0}: Error finding container a7142c6baaaef9f5045860bc1fad336509b3c333ff1d32d0ee533ce079af8af5: Status 404 returned error can't find the container with id a7142c6baaaef9f5045860bc1fad336509b3c333ff1d32d0ee533ce079af8af5
Oct 31 23:09:55.237587 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:55.237528    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"]
Oct 31 23:09:55.238427 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:55.238304    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:09:55.238427 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:55.238399    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:09:55.238591 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:55.238444    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:09:55.301712 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:55.301674    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm" event={"ID":"d361d757-5e4e-43a4-acb7-9ef5f24113af","Type":"ContainerStarted","Data":"8492f0084fcae522ce2b7c039dd6b8feba4000a065cae3671ef280a4218c4068"}
Oct 31 23:09:55.302133 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:55.301732    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm" event={"ID":"d361d757-5e4e-43a4-acb7-9ef5f24113af","Type":"ContainerStarted","Data":"a400b386ef625053d3ae414a6f86fd02b36eb663691a2b826940a12f08b9944f"}
Oct 31 23:09:55.304659 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:55.304581    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq" event={"ID":"cdcadd41-cef3-4fbc-9551-ed8dae06f790","Type":"ContainerStarted","Data":"d1a3317cd014ecdb689a2e97406f15a7370239b8ddf845d0a4eb845546793a86"}
Oct 31 23:09:55.304659 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:55.304629    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq" event={"ID":"cdcadd41-cef3-4fbc-9551-ed8dae06f790","Type":"ContainerStarted","Data":"152b69f50aad27a1f7c30d13631cff355b04e1f958da09ffdda0e9bec758bde7"}
Oct 31 23:09:55.306243 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:55.306178    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t" event={"ID":"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5","Type":"ContainerStarted","Data":"a7142c6baaaef9f5045860bc1fad336509b3c333ff1d32d0ee533ce079af8af5"}
Oct 31 23:09:55.308180 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:55.307761    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2" event={"ID":"f421d513-57d6-43b1-af00-bed486a4d0bf","Type":"ContainerStarted","Data":"26c66833d17c4de159bed0375522a142211248db1eb912d9c40bbce7f7f97004"}
Oct 31 23:09:55.308180 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:55.307794    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2" event={"ID":"f421d513-57d6-43b1-af00-bed486a4d0bf","Type":"ContainerStarted","Data":"c66e6e07094b11a4e0cafe82215a981a290427f1d7e057e15fe4a9850c380f30"}
Oct 31 23:09:55.337575 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:55.336503    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm" podStartSLOduration=1.336483248 podStartE2EDuration="1.336483248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:09:55.336120789 +0000 UTC m=+1214.083310296" watchObservedRunningTime="2026-10-31 23:09:55.336483248 +0000 UTC m=+1214.083672754"
Oct 31 23:09:55.402375 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:55.401258    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2" podStartSLOduration=1.401241374 podStartE2EDuration="1.401241374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:09:55.400497807 +0000 UTC m=+1214.147687315" watchObservedRunningTime="2026-10-31 23:09:55.401241374 +0000 UTC m=+1214.148430878"
Oct 31 23:09:55.402375 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:55.401456    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq" podStartSLOduration=1.401448255 podStartE2EDuration="1.401448255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:09:55.364054569 +0000 UTC m=+1214.111244076" watchObservedRunningTime="2026-10-31 23:09:55.401448255 +0000 UTC m=+1214.148637761"
Oct 31 23:09:55.631463 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:55.626479    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:09:55.631463 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:55.631375    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:09:55.810135 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:55.810100    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:09:56.344972 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:56.344939    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t" event={"ID":"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5","Type":"ContainerStarted","Data":"5494c25f1a03d3cbc311c4194963bf23987c9df812cb86599e8fde7524ff4f2d"}
Oct 31 23:09:56.644383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:56.642967    4301 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 31 23:09:56.644383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:56.643031    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq" podUID="cdcadd41-cef3-4fbc-9551-ed8dae06f790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 31 23:09:56.648885 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:56.648850    4301 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 31 23:09:56.649106 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:56.649077    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2" podUID="f421d513-57d6-43b1-af00-bed486a4d0bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 31 23:09:56.812378 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:56.811344    4301 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 31 23:09:56.812378 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:56.811424    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm" podUID="d361d757-5e4e-43a4-acb7-9ef5f24113af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 31 23:09:56.938675 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:56.938583    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:09:56.938993 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:56.938973    4301 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 31 23:09:56.939152 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:56.939129    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t" podUID="fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 31 23:09:57.628844 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:57.628805    4301 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 31 23:09:57.630462 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:57.630432    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq" podUID="cdcadd41-cef3-4fbc-9551-ed8dae06f790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 31 23:09:57.630920 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:57.630342    4301 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 31 23:09:57.631098 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:57.631077    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2" podUID="f421d513-57d6-43b1-af00-bed486a4d0bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 31 23:09:57.814383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:57.813753    4301 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 31 23:09:57.814383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:57.813814    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm" podUID="d361d757-5e4e-43a4-acb7-9ef5f24113af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 31 23:09:57.936458 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:57.935859    4301 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 31 23:09:57.936458 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:57.935933    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t" podUID="fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 31 23:09:58.153383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:58.142661    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:09:58.155202 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:58.155163    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:09:58.650284 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:58.650248    4301 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 31 23:09:58.654697 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:58.651035    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq" podUID="cdcadd41-cef3-4fbc-9551-ed8dae06f790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 31 23:09:58.654977 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:58.654894    4301 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 31 23:09:58.655134 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:58.655113    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2" podUID="f421d513-57d6-43b1-af00-bed486a4d0bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 31 23:09:58.809811 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:58.809774    4301 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 31 23:09:58.810409 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:58.810051    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm" podUID="d361d757-5e4e-43a4-acb7-9ef5f24113af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 31 23:09:58.937881 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:58.937768    4301 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 31 23:09:58.938157 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:58.938111    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t" podUID="fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 31 23:09:59.629829 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:59.629787    4301 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 31 23:09:59.630100 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:59.630070    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq" podUID="cdcadd41-cef3-4fbc-9551-ed8dae06f790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 31 23:09:59.631527 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:59.631502    4301 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 31 23:09:59.631755 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:59.631733    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2" podUID="f421d513-57d6-43b1-af00-bed486a4d0bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 31 23:09:59.809937 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:59.809900    4301 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 31 23:09:59.810586 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:59.810539    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm" podUID="d361d757-5e4e-43a4-acb7-9ef5f24113af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 31 23:09:59.939608 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:59.939504    4301 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 31 23:09:59.939896 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:09:59.939847    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t" podUID="fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 31 23:10:00.630379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:00.629870    4301 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 31 23:10:00.630379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:00.629947    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq" podUID="cdcadd41-cef3-4fbc-9551-ed8dae06f790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 31 23:10:00.634376 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:00.630680    4301 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 31 23:10:00.634376 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:00.630741    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2" podUID="f421d513-57d6-43b1-af00-bed486a4d0bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 31 23:10:00.810258 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:00.810208    4301 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 31 23:10:00.810887 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:00.810838    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm" podUID="d361d757-5e4e-43a4-acb7-9ef5f24113af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 31 23:10:00.943439 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:00.942485    4301 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 31 23:10:00.943439 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:00.942559    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t" podUID="fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 31 23:10:01.065383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:01.060553    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 31 23:10:01.098624 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:01.098484    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:10:01.143392 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:01.142505    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:10:01.191382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:01.189515    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:10:01.631380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:01.627563    4301 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 31 23:10:01.631380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:01.627637    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq" podUID="cdcadd41-cef3-4fbc-9551-ed8dae06f790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 31 23:10:01.631380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:01.628213    4301 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 31 23:10:01.631380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:01.628422    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2" podUID="f421d513-57d6-43b1-af00-bed486a4d0bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 31 23:10:01.815384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:01.812783    4301 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 31 23:10:01.815384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:01.812849    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm" podUID="d361d757-5e4e-43a4-acb7-9ef5f24113af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 31 23:10:01.939435 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:01.936801    4301 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 31 23:10:01.939435 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:01.936864    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t" podUID="fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 31 23:10:02.065385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:02.064473    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:10:02.065385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:02.065119    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:10:02.095383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:02.094513    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t" podStartSLOduration=8.094492248 podStartE2EDuration="8.094492248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:09:56.379198169 +0000 UTC m=+1215.126387676" watchObservedRunningTime="2026-10-31 23:10:02.094492248 +0000 UTC m=+1220.841681757"
Oct 31 23:10:02.101396 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:02.100505    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:10:02.101396 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:02.101379    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:10:02.311779 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:02.148987    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:10:02.311779 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:02.150934    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:10:02.311779 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:02.195510    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 23:10:02.311779 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:02.196132    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:10:02.367227 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:02.329328    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:10:02.592487 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:02.569869    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:10:02.662686 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:02.633526    4301 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 31 23:10:02.662686 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:02.662289    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq" podUID="cdcadd41-cef3-4fbc-9551-ed8dae06f790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 31 23:10:02.664690 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:02.664668    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:10:02.671696 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:02.671665    4301 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 31 23:10:02.679481 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:02.679442    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2" podUID="f421d513-57d6-43b1-af00-bed486a4d0bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 31 23:10:02.736458 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:02.736134    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:10:02.793892 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:10:02.793858    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:10:02.813517 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:02.813484    4301 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 31 23:10:02.813782 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:02.813753    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm" podUID="d361d757-5e4e-43a4-acb7-9ef5f24113af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 31 23:10:02.935771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:02.935732    4301 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 31 23:10:02.936302 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:02.935800    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t" podUID="fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 31 23:10:03.635249 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:03.635210    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:10:03.636273 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:03.636248    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:10:03.637573 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:03.637549    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:10:03.814071 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:03.814038    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:10:03.814415 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:03.814388    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:10:03.815186 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:03.815168    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:10:03.900382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:03.900275    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:10:03.901724 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:03.901700    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:10:03.902912 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:03.902887    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:10:03.939188 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:03.939160    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:10:03.939630 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:03.939462    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:10:03.940092 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:03.940078    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:10:07.895783 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:07.895752    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:10:09.072382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.066876    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"]
Oct 31 23:10:09.083380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.078730    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:10:09.083380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.082926    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"]
Oct 31 23:10:09.107410 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.095064    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-04c5c2-openshift-default-dockercfg-js5sc\""
Oct 31 23:10:09.290383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.287152    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-workload-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:10:09.290383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.287212    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-credential-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:10:09.290383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.287265    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istio-token\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:10:09.290383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.287313    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istio-envoy\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:10:09.299380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.287349    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-workload-certs\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:10:09.299380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.297093    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istio-podinfo\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:10:09.299380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.297134    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmr8j\" (UniqueName: \"kubernetes.io/projected/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-kube-api-access-pmr8j\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:10:09.299380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.297206    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istio-data\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:10:09.299380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.297271    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istiod-ca-cert\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:10:09.420381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.402167    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-workload-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:10:09.420381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.402219    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-credential-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:10:09.420381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.402267    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istio-token\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:10:09.420381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.402316    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istio-envoy\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:10:09.420381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.402372    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-workload-certs\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:10:09.420381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.402411    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istio-podinfo\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:10:09.420381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.402444    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmr8j\" (UniqueName: \"kubernetes.io/projected/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-kube-api-access-pmr8j\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:10:09.420381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.402503    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istio-data\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:10:09.420381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.402556    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istiod-ca-cert\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:10:09.420381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.403339    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istiod-ca-cert\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:10:09.420381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.403768    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-workload-certs\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:10:09.420381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.404081    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istio-data\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:10:09.420381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.406185    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-workload-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:10:09.420381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.408329    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-credential-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:10:09.420381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.418988    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istio-token\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:10:09.420381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.419316    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmr8j\" (UniqueName: \"kubernetes.io/projected/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-kube-api-access-pmr8j\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:10:09.420381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.419571    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istio-envoy\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:10:09.421447 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.420596    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istio-podinfo\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:10:09.441699 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.441656    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:10:09.731641 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.731278    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"]
Oct 31 23:10:09.747595 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:10:09.747564    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84b0a9a3_a72d_4a7e_8b45_db92fe46a6f3.slice/crio-2ed2e7fbd8636b6d866b9d8d947c4d733b492956b208723aa26662490b22e8ac WatchSource:0}: Error finding container 2ed2e7fbd8636b6d866b9d8d947c4d733b492956b208723aa26662490b22e8ac: Status 404 returned error can't find the container with id 2ed2e7fbd8636b6d866b9d8d947c4d733b492956b208723aa26662490b22e8ac
Oct 31 23:10:09.753687 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.753630    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:10:09.754234 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.754205    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:10:09.755234 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:09.755194    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:10:10.662055 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:10.662020    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn" event={"ID":"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3","Type":"ContainerStarted","Data":"2c6dbfc4df937412f010f1576feb072186ded77215d6407db82849e339eb2b97"}
Oct 31 23:10:10.662467 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:10.662080    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn" event={"ID":"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3","Type":"ContainerStarted","Data":"2ed2e7fbd8636b6d866b9d8d947c4d733b492956b208723aa26662490b22e8ac"}
Oct 31 23:10:10.688120 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:10.688074    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn" podStartSLOduration=1.688060563 podStartE2EDuration="1.688060563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:10:10.68586629 +0000 UTC m=+1229.433055924" watchObservedRunningTime="2026-10-31 23:10:10.688060563 +0000 UTC m=+1229.435250148"
Oct 31 23:10:11.449381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:11.444748    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:10:11.453504 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:11.453474    4301 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 31 23:10:11.453650 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:11.453557    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn" podUID="84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 31 23:10:12.446120 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:12.445780    4301 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 31 23:10:12.446120 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:12.445857    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn" podUID="84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 31 23:10:13.442721 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:13.442685    4301 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 31 23:10:13.442991 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:13.442964    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn" podUID="84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 31 23:10:14.446382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:14.445753    4301 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 31 23:10:14.446382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:14.445835    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn" podUID="84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 31 23:10:15.450383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:15.448824    4301 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 31 23:10:15.450383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:15.448889    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn" podUID="84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 31 23:10:16.449428 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:16.449397    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:10:16.450105 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:16.450083    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:10:16.451061 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:16.451034    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:10:24.264106 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:24.264064    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"]
Oct 31 23:10:24.269101 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:24.269077    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:10:24.364344 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:24.364290    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"]
Oct 31 23:10:24.373105 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:24.370676    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a16dfb-openshift-default-dockercfg-lchgz\""
Oct 31 23:10:24.373105 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:24.371482    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-workload-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:10:24.373105 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:24.371533    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-credential-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:10:24.373105 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:24.371567    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-istio-data\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:10:24.373105 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:24.371599    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d3c8af1-7e81-43b7-998c-6ba875936720-istiod-ca-cert\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:10:24.373105 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:24.371656    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d3c8af1-7e81-43b7-998c-6ba875936720-istio-token\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:10:24.373105 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:24.371734    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d3c8af1-7e81-43b7-998c-6ba875936720-istio-podinfo\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:10:24.373105 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:24.371781    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-workload-certs\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:10:24.373105 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:24.371817    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp7cr\" (UniqueName: \"kubernetes.io/projected/5d3c8af1-7e81-43b7-998c-6ba875936720-kube-api-access-sp7cr\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:10:24.373105 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:24.371844    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-istio-envoy\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:10:24.473102 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:24.473066    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-workload-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:10:24.473274 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:24.473118    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-credential-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:10:24.473274 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:24.473148    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-istio-data\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:10:24.473274 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:24.473179    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d3c8af1-7e81-43b7-998c-6ba875936720-istiod-ca-cert\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:10:24.473274 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:24.473232    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d3c8af1-7e81-43b7-998c-6ba875936720-istio-token\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:10:24.473487 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:24.473305    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d3c8af1-7e81-43b7-998c-6ba875936720-istio-podinfo\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:10:24.473487 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:24.473385    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-workload-certs\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:10:24.473487 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:24.473420    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp7cr\" (UniqueName: \"kubernetes.io/projected/5d3c8af1-7e81-43b7-998c-6ba875936720-kube-api-access-sp7cr\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:10:24.473487 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:24.473449    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-istio-envoy\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:10:24.473718 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:24.473560    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-workload-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:10:24.473718 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:24.473632    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-credential-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:10:24.474061 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:24.474035    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d3c8af1-7e81-43b7-998c-6ba875936720-istiod-ca-cert\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:10:24.474151 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:24.474050    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-workload-certs\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:10:24.474267 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:24.474251    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-istio-data\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:10:24.475746 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:24.475722    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-istio-envoy\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:10:24.475930 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:24.475913    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d3c8af1-7e81-43b7-998c-6ba875936720-istio-podinfo\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:10:24.499251 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:24.499218    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d3c8af1-7e81-43b7-998c-6ba875936720-istio-token\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:10:24.501229 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:24.501204    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp7cr\" (UniqueName: \"kubernetes.io/projected/5d3c8af1-7e81-43b7-998c-6ba875936720-kube-api-access-sp7cr\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:10:24.690207 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:24.690160    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:10:25.048107 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:25.048047    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:10:25.048243 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:25.048170    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:10:25.048243 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:25.048235    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:10:25.056439 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:25.054181    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"]
Oct 31 23:10:25.763202 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:25.763159    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g" event={"ID":"5d3c8af1-7e81-43b7-998c-6ba875936720","Type":"ContainerStarted","Data":"05caa5d59190e2005628db6cec4e139e099aa7abadc4edc23a3f3c201fc9725e"}
Oct 31 23:10:25.763635 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:25.763231    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g" event={"ID":"5d3c8af1-7e81-43b7-998c-6ba875936720","Type":"ContainerStarted","Data":"4af14cf85ab5edbf21137c2e6f7bc304551ba089300b837c70fe0bd853a22cbe"}
Oct 31 23:10:25.803409 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:25.796444    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g" podStartSLOduration=1.796418464 podStartE2EDuration="1.796418464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:10:25.790147121 +0000 UTC m=+1244.537336627" watchObservedRunningTime="2026-10-31 23:10:25.796418464 +0000 UTC m=+1244.543607970"
Oct 31 23:10:26.691433 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:26.691397    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:10:26.693475 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:26.693447    4301 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 31 23:10:26.693632 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:26.693506    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g" podUID="5d3c8af1-7e81-43b7-998c-6ba875936720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 31 23:10:27.691343 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:27.691308    4301 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 31 23:10:27.691853 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:27.691384    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g" podUID="5d3c8af1-7e81-43b7-998c-6ba875936720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 31 23:10:28.691343 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:28.691307    4301 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 31 23:10:28.691770 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:28.691387    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g" podUID="5d3c8af1-7e81-43b7-998c-6ba875936720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 31 23:10:29.691964 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:29.691923    4301 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 31 23:10:29.692503 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:29.691994    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g" podUID="5d3c8af1-7e81-43b7-998c-6ba875936720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 31 23:10:30.691282 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:30.691252    4301 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 31 23:10:30.691470 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:30.691303    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g" podUID="5d3c8af1-7e81-43b7-998c-6ba875936720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 31 23:10:31.704897 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:31.704853    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:10:31.705422 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:31.705398    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:10:31.706648 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:31.706620    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:10:32.545698 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:10:32.545660    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:10:33.190182 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:33.190139    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:10:54.592652 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:54.592616    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65138e65-4x275"]
Oct 31 23:10:54.597454 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:54.597436    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65138e65-4x275"
Oct 31 23:10:54.604552 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:54.604526    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 23:10:54.604963 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:54.604940    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\""
Oct 31 23:10:54.607741 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:54.607715    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65138e65-4x275"]
Oct 31 23:10:54.669680 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:54.669286    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbcf409c-242f-4f2c-ad59-c320f6e27af0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65138e65-4x275\" (UID: \"cbcf409c-242f-4f2c-ad59-c320f6e27af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65138e65-4x275"
Oct 31 23:10:54.669680 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:54.669342    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npqxw\" (UniqueName: \"kubernetes.io/projected/cbcf409c-242f-4f2c-ad59-c320f6e27af0-kube-api-access-npqxw\") pod \"maas-api-revoke-sub-65138e65-4x275\" (UID: \"cbcf409c-242f-4f2c-ad59-c320f6e27af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65138e65-4x275"
Oct 31 23:10:54.669680 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:54.669516    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbcf409c-242f-4f2c-ad59-c320f6e27af0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65138e65-4x275\" (UID: \"cbcf409c-242f-4f2c-ad59-c320f6e27af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65138e65-4x275"
Oct 31 23:10:54.770321 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:54.770290    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbcf409c-242f-4f2c-ad59-c320f6e27af0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65138e65-4x275\" (UID: \"cbcf409c-242f-4f2c-ad59-c320f6e27af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65138e65-4x275"
Oct 31 23:10:54.770503 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:54.770395    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbcf409c-242f-4f2c-ad59-c320f6e27af0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65138e65-4x275\" (UID: \"cbcf409c-242f-4f2c-ad59-c320f6e27af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65138e65-4x275"
Oct 31 23:10:54.770503 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:54.770420    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npqxw\" (UniqueName: \"kubernetes.io/projected/cbcf409c-242f-4f2c-ad59-c320f6e27af0-kube-api-access-npqxw\") pod \"maas-api-revoke-sub-65138e65-4x275\" (UID: \"cbcf409c-242f-4f2c-ad59-c320f6e27af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65138e65-4x275"
Oct 31 23:10:54.771035 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:54.771013    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbcf409c-242f-4f2c-ad59-c320f6e27af0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65138e65-4x275\" (UID: \"cbcf409c-242f-4f2c-ad59-c320f6e27af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65138e65-4x275"
Oct 31 23:10:54.772776 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:54.772753    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbcf409c-242f-4f2c-ad59-c320f6e27af0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65138e65-4x275\" (UID: \"cbcf409c-242f-4f2c-ad59-c320f6e27af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65138e65-4x275"
Oct 31 23:10:54.781916 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:54.781886    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npqxw\" (UniqueName: \"kubernetes.io/projected/cbcf409c-242f-4f2c-ad59-c320f6e27af0-kube-api-access-npqxw\") pod \"maas-api-revoke-sub-65138e65-4x275\" (UID: \"cbcf409c-242f-4f2c-ad59-c320f6e27af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65138e65-4x275"
Oct 31 23:10:54.911537 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:54.911504    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65138e65-4x275"
Oct 31 23:10:55.190706 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:55.190682    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65138e65-4x275"]
Oct 31 23:10:55.193112 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:10:55.193081    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbcf409c_242f_4f2c_ad59_c320f6e27af0.slice/crio-35e4390c082041e0202ded914504d1fd7ea596ca8accdb2619807cc49a396eea WatchSource:0}: Error finding container 35e4390c082041e0202ded914504d1fd7ea596ca8accdb2619807cc49a396eea: Status 404 returned error can't find the container with id 35e4390c082041e0202ded914504d1fd7ea596ca8accdb2619807cc49a396eea
Oct 31 23:10:55.195082 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:55.195063    4301 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 23:10:55.977718 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:55.977679    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65138e65-4x275" event={"ID":"cbcf409c-242f-4f2c-ad59-c320f6e27af0","Type":"ContainerStarted","Data":"35e4390c082041e0202ded914504d1fd7ea596ca8accdb2619807cc49a396eea"}
Oct 31 23:10:58.993810 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:58.993778    4301 generic.go:356] "Generic (PLEG): container finished" podID="cbcf409c-242f-4f2c-ad59-c320f6e27af0" containerID="71b6cea34e5e3c11f62f9d08160dbff715a8efa603dcddcc8c425e9aa5b1356b" exitCode=0
Oct 31 23:10:58.994210 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:58.993861    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65138e65-4x275" event={"ID":"cbcf409c-242f-4f2c-ad59-c320f6e27af0","Type":"ContainerDied","Data":"71b6cea34e5e3c11f62f9d08160dbff715a8efa603dcddcc8c425e9aa5b1356b"}
Oct 31 23:10:59.125294 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:10:59.125256    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:11:01.029609 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:01.029575    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65138e65-4x275"
Oct 31 23:11:01.049904 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:01.049867    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65138e65-4x275" event={"ID":"cbcf409c-242f-4f2c-ad59-c320f6e27af0","Type":"ContainerDied","Data":"35e4390c082041e0202ded914504d1fd7ea596ca8accdb2619807cc49a396eea"}
Oct 31 23:11:01.050283 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:01.050263    4301 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35e4390c082041e0202ded914504d1fd7ea596ca8accdb2619807cc49a396eea"
Oct 31 23:11:01.050459 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:01.050194    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65138e65-4x275"
Oct 31 23:11:01.054991 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:01.054955    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbcf409c-242f-4f2c-ad59-c320f6e27af0-kube-api-access-npqxw\" (UniqueName: \"kubernetes.io/projected/cbcf409c-242f-4f2c-ad59-c320f6e27af0-kube-api-access-npqxw\") pod \"cbcf409c-242f-4f2c-ad59-c320f6e27af0\" (UID: \"cbcf409c-242f-4f2c-ad59-c320f6e27af0\") "
Oct 31 23:11:01.055280 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:01.055260    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbcf409c-242f-4f2c-ad59-c320f6e27af0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbcf409c-242f-4f2c-ad59-c320f6e27af0-maas-api-xks-ca\") pod \"cbcf409c-242f-4f2c-ad59-c320f6e27af0\" (UID: \"cbcf409c-242f-4f2c-ad59-c320f6e27af0\") "
Oct 31 23:11:01.055548 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:01.055529    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbcf409c-242f-4f2c-ad59-c320f6e27af0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbcf409c-242f-4f2c-ad59-c320f6e27af0-maas-api-openshift-ca\") pod \"cbcf409c-242f-4f2c-ad59-c320f6e27af0\" (UID: \"cbcf409c-242f-4f2c-ad59-c320f6e27af0\") "
Oct 31 23:11:01.059619 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:01.059579    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbcf409c-242f-4f2c-ad59-c320f6e27af0-maas-api-openshift-ca" pod "cbcf409c-242f-4f2c-ad59-c320f6e27af0" (UID: "cbcf409c-242f-4f2c-ad59-c320f6e27af0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:11:01.085809 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:01.085758    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbcf409c-242f-4f2c-ad59-c320f6e27af0-kube-api-access-npqxw" pod "cbcf409c-242f-4f2c-ad59-c320f6e27af0" (UID: "cbcf409c-242f-4f2c-ad59-c320f6e27af0"). InnerVolumeSpecName "kube-api-access-npqxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:11:01.095735 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:01.095700    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbcf409c-242f-4f2c-ad59-c320f6e27af0-maas-api-xks-ca" pod "cbcf409c-242f-4f2c-ad59-c320f6e27af0" (UID: "cbcf409c-242f-4f2c-ad59-c320f6e27af0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:11:01.178867 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:01.178754    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbcf409c-242f-4f2c-ad59-c320f6e27af0-maas-api-openshift-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:11:01.179118 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:01.179096    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npqxw\" (UniqueName: \"kubernetes.io/projected/cbcf409c-242f-4f2c-ad59-c320f6e27af0-kube-api-access-npqxw\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:11:01.179254 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:01.179238    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbcf409c-242f-4f2c-ad59-c320f6e27af0-maas-api-xks-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:11:01.270535 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:01.270477    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:11:01.337547 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:01.337489    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 23:11:02.622580 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:11:02.622532    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:11:15.497415 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:15.497348    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"]
Oct 31 23:11:15.498132 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:15.498030    4301 cpu_manager.go:414] "containerMap: removing container" podUID="cbcf409c-242f-4f2c-ad59-c320f6e27af0" containerName="revoke-keys"
Oct 31 23:11:15.498132 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:15.498046    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbcf409c-242f-4f2c-ad59-c320f6e27af0" containerName="revoke-keys"
Oct 31 23:11:15.498132 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:15.498128    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbcf409c-242f-4f2c-ad59-c320f6e27af0" containerName="revoke-keys"
Oct 31 23:11:15.501919 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:15.501899    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:11:15.510670 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:15.510646    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-04c5c2\"/\"kube-root-ca.crt\""
Oct 31 23:11:15.510932 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:15.510916    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-04c5c2\"/\"openshift-service-ca.crt\""
Oct 31 23:11:15.511026 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:15.510991    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-04c5c2\"/\"distinct-w1-04c5c2-kserve-self-signed-certs\""
Oct 31 23:11:15.511167 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:15.511150    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-04c5c2\"/\"default-dockercfg-jcdwr\""
Oct 31 23:11:15.513548 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:15.513523    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"]
Oct 31 23:11:15.539652 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:15.539596    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c95b\" (UniqueName: \"kubernetes.io/projected/570b97e3-1e23-474c-beb7-9438692c5508-kube-api-access-5c95b\") pod \"distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd\" (UID: \"570b97e3-1e23-474c-beb7-9438692c5508\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:11:15.539831 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:15.539807    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/570b97e3-1e23-474c-beb7-9438692c5508-tmp-dir\") pod \"distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd\" (UID: \"570b97e3-1e23-474c-beb7-9438692c5508\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:11:15.539920 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:15.539902    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/570b97e3-1e23-474c-beb7-9438692c5508-dshm\") pod \"distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd\" (UID: \"570b97e3-1e23-474c-beb7-9438692c5508\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:11:15.539961 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:15.539942    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/570b97e3-1e23-474c-beb7-9438692c5508-home\") pod \"distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd\" (UID: \"570b97e3-1e23-474c-beb7-9438692c5508\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:11:15.540007 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:15.539966    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/570b97e3-1e23-474c-beb7-9438692c5508-tls-certs\") pod \"distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd\" (UID: \"570b97e3-1e23-474c-beb7-9438692c5508\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:11:15.540007 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:15.539993    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/570b97e3-1e23-474c-beb7-9438692c5508-model-cache\") pod \"distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd\" (UID: \"570b97e3-1e23-474c-beb7-9438692c5508\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:11:15.644384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:15.641700    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/570b97e3-1e23-474c-beb7-9438692c5508-tmp-dir\") pod \"distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd\" (UID: \"570b97e3-1e23-474c-beb7-9438692c5508\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:11:15.644384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:15.641822    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/570b97e3-1e23-474c-beb7-9438692c5508-dshm\") pod \"distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd\" (UID: \"570b97e3-1e23-474c-beb7-9438692c5508\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:11:15.644384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:15.641866    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/570b97e3-1e23-474c-beb7-9438692c5508-home\") pod \"distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd\" (UID: \"570b97e3-1e23-474c-beb7-9438692c5508\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:11:15.644384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:15.641890    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/570b97e3-1e23-474c-beb7-9438692c5508-tls-certs\") pod \"distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd\" (UID: \"570b97e3-1e23-474c-beb7-9438692c5508\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:11:15.644384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:15.641917    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/570b97e3-1e23-474c-beb7-9438692c5508-model-cache\") pod \"distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd\" (UID: \"570b97e3-1e23-474c-beb7-9438692c5508\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:11:15.644384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:15.641976    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c95b\" (UniqueName: \"kubernetes.io/projected/570b97e3-1e23-474c-beb7-9438692c5508-kube-api-access-5c95b\") pod \"distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd\" (UID: \"570b97e3-1e23-474c-beb7-9438692c5508\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:11:15.644384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:15.643067    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/570b97e3-1e23-474c-beb7-9438692c5508-tmp-dir\") pod \"distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd\" (UID: \"570b97e3-1e23-474c-beb7-9438692c5508\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:11:15.644384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:15.643777    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/570b97e3-1e23-474c-beb7-9438692c5508-model-cache\") pod \"distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd\" (UID: \"570b97e3-1e23-474c-beb7-9438692c5508\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:11:15.644384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:15.644006    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/570b97e3-1e23-474c-beb7-9438692c5508-home\") pod \"distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd\" (UID: \"570b97e3-1e23-474c-beb7-9438692c5508\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:11:15.684404 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:15.678199    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/570b97e3-1e23-474c-beb7-9438692c5508-tls-certs\") pod \"distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd\" (UID: \"570b97e3-1e23-474c-beb7-9438692c5508\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:11:15.704224 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:15.702777    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/570b97e3-1e23-474c-beb7-9438692c5508-dshm\") pod \"distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd\" (UID: \"570b97e3-1e23-474c-beb7-9438692c5508\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:11:15.704224 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:15.703479    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c95b\" (UniqueName: \"kubernetes.io/projected/570b97e3-1e23-474c-beb7-9438692c5508-kube-api-access-5c95b\") pod \"distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd\" (UID: \"570b97e3-1e23-474c-beb7-9438692c5508\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:11:15.815597 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:15.815513    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:11:16.094056 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:16.093947    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"]
Oct 31 23:11:16.096931 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:11:16.096898    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod570b97e3_1e23_474c_beb7_9438692c5508.slice/crio-81768cb8b32f79b624805fdaf684efbb902b9436ae9568118f0095c1bb4654bc WatchSource:0}: Error finding container 81768cb8b32f79b624805fdaf684efbb902b9436ae9568118f0095c1bb4654bc: Status 404 returned error can't find the container with id 81768cb8b32f79b624805fdaf684efbb902b9436ae9568118f0095c1bb4654bc
Oct 31 23:11:16.203962 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:16.203831    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" event={"ID":"570b97e3-1e23-474c-beb7-9438692c5508","Type":"ContainerStarted","Data":"8fb839d012c9f4a9840c24ad3e5fce6c480abe9ead065bcefa09a2322af595e3"}
Oct 31 23:11:16.203962 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:16.203880    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" event={"ID":"570b97e3-1e23-474c-beb7-9438692c5508","Type":"ContainerStarted","Data":"81768cb8b32f79b624805fdaf684efbb902b9436ae9568118f0095c1bb4654bc"}
Oct 31 23:11:16.220801 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:16.220751    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podStartSLOduration=1.2207346829999999 podStartE2EDuration="1.220734683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:11:16.220688234 +0000 UTC m=+1294.967877742" watchObservedRunningTime="2026-10-31 23:11:16.220734683 +0000 UTC m=+1294.967924200"
Oct 31 23:11:16.816599 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:16.816559    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:11:24.366124 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.366036    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"]
Oct 31 23:11:24.370029 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.370009    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:11:24.385105 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.385081    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"]
Oct 31 23:11:24.392988 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.392960    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzfqk\" (UniqueName: \"kubernetes.io/projected/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-kube-api-access-kzfqk\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:11:24.393124 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.393001    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-workload-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:11:24.393124 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.393026    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istio-token\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:11:24.393211 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.393174    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istio-data\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:11:24.393267 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.393222    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-workload-certs\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:11:24.393326 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.393280    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istio-envoy\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:11:24.393326 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.393297    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istio-podinfo\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:11:24.393430 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.393349    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-credential-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:11:24.393497 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.393469    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istiod-ca-cert\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:11:24.423459 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.423424    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"]
Oct 31 23:11:24.427066 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.427048    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:11:24.440586 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.440564    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"]
Oct 31 23:11:24.502098 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.502061    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istio-envoy\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:11:24.502098 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.502096    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istio-podinfo\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:11:24.502442 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.502126    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-credential-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w8722\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:11:24.502442 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.502148    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istio-data\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w8722\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:11:24.502442 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.502206    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-credential-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:11:24.502442 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.502276    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-workload-certs\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w8722\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:11:24.502669 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.502548    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istiod-ca-cert\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:11:24.502669 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.502609    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-credential-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:11:24.502669 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.502622    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzfqk\" (UniqueName: \"kubernetes.io/projected/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-kube-api-access-kzfqk\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:11:24.502822 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.502668    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-workload-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:11:24.502822 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.502711    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istio-token\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:11:24.502822 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.502751    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-workload-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w8722\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:11:24.502822 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.502784    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istio-token\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w8722\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:11:24.502822 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.502811    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istio-envoy\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w8722\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:11:24.503068 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.502852    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k59xn\" (UniqueName: \"kubernetes.io/projected/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-kube-api-access-k59xn\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w8722\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:11:24.503068 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.502887    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istio-data\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:11:24.503068 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.502912    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istiod-ca-cert\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w8722\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:11:24.503220 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.503075    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-workload-certs\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:11:24.503220 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.503102    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istio-podinfo\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w8722\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:11:24.503220 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.503168    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istiod-ca-cert\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:11:24.503400 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.503347    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istio-data\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:11:24.503457 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.503409    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-workload-certs\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:11:24.505241 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.505212    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istio-envoy\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:11:24.505331 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.505246    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istio-podinfo\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:11:24.505415 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.505405    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-workload-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:11:24.514499 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.514478    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istio-token\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:11:24.514712 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.514691    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzfqk\" (UniqueName: \"kubernetes.io/projected/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-kube-api-access-kzfqk\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:11:24.610194 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.607709    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-workload-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w8722\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:11:24.610194 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.607750    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istio-token\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w8722\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:11:24.610194 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.607779    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istio-envoy\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w8722\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:11:24.610194 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.607824    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k59xn\" (UniqueName: \"kubernetes.io/projected/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-kube-api-access-k59xn\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w8722\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:11:24.610194 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.607861    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istiod-ca-cert\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w8722\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:11:24.610194 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.607902    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istio-podinfo\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w8722\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:11:24.610194 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.607944    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-credential-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w8722\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:11:24.610194 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.607959    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istio-data\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w8722\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:11:24.610194 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.607995    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-workload-certs\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w8722\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:11:24.610194 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.608450    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-workload-certs\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w8722\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:11:24.610194 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.608684    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-workload-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w8722\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:11:24.610194 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.609818    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-credential-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w8722\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:11:24.611057 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.610378    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istio-data\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w8722\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:11:24.611442 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.611415    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istiod-ca-cert\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w8722\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:11:24.615019 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.614992    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istio-podinfo\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w8722\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:11:24.616009 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.615984    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istio-envoy\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w8722\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:11:24.623057 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.623017    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istio-token\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w8722\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:11:24.623609 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.623585    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k59xn\" (UniqueName: \"kubernetes.io/projected/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-kube-api-access-k59xn\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w8722\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:11:24.684582 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.684548    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:11:24.741131 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:24.741093    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:11:25.006708 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:25.006683    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"]
Oct 31 23:11:25.008279 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:11:25.008250    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde434cd6_3610_47ac_ae7e_78e1d05cb0c5.slice/crio-38d85dce31df8e57567143370194c087cc263b08e73ccb840c072ee7a6ab70c2 WatchSource:0}: Error finding container 38d85dce31df8e57567143370194c087cc263b08e73ccb840c072ee7a6ab70c2: Status 404 returned error can't find the container with id 38d85dce31df8e57567143370194c087cc263b08e73ccb840c072ee7a6ab70c2
Oct 31 23:11:25.010628 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:25.010592    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:11:25.010742 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:25.010660    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:11:25.010742 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:25.010689    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:11:25.067278 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:25.067254    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"]
Oct 31 23:11:25.068514 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:11:25.068486    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff5f8cb0_0541_4dae_9a86_8c3c0fdda126.slice/crio-776e0e0e54c588f184213be813c27164cc619bd01760f1fee79311bd96948e8b WatchSource:0}: Error finding container 776e0e0e54c588f184213be813c27164cc619bd01760f1fee79311bd96948e8b: Status 404 returned error can't find the container with id 776e0e0e54c588f184213be813c27164cc619bd01760f1fee79311bd96948e8b
Oct 31 23:11:25.071149 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:25.071114    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:11:25.071250 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:25.071186    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:11:25.071250 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:25.071213    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:11:25.361104 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:25.361013    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722" event={"ID":"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126","Type":"ContainerStarted","Data":"f8d872a7669672df63aab7645e0c3a3a5e825e5c5c452146ddcda048667201b9"}
Oct 31 23:11:25.361104 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:25.361092    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722" event={"ID":"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126","Type":"ContainerStarted","Data":"776e0e0e54c588f184213be813c27164cc619bd01760f1fee79311bd96948e8b"}
Oct 31 23:11:25.362624 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:25.362599    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm" event={"ID":"de434cd6-3610-47ac-ae7e-78e1d05cb0c5","Type":"ContainerStarted","Data":"8342fb6cfa9189bbefe86a43bbe1878ce7caae656611eeb74aac34a0ee189a9c"}
Oct 31 23:11:25.362736 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:25.362642    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm" event={"ID":"de434cd6-3610-47ac-ae7e-78e1d05cb0c5","Type":"ContainerStarted","Data":"38d85dce31df8e57567143370194c087cc263b08e73ccb840c072ee7a6ab70c2"}
Oct 31 23:11:25.384615 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:25.384548    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722" podStartSLOduration=1.38453445 podStartE2EDuration="1.38453445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:11:25.382250753 +0000 UTC m=+1304.129440257" watchObservedRunningTime="2026-10-31 23:11:25.38453445 +0000 UTC m=+1304.131723956"
Oct 31 23:11:25.404888 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:25.404831    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm" podStartSLOduration=1.404815805 podStartE2EDuration="1.404815805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:11:25.402469969 +0000 UTC m=+1304.149659476" watchObservedRunningTime="2026-10-31 23:11:25.404815805 +0000 UTC m=+1304.152005311"
Oct 31 23:11:25.685624 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:25.685580    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:11:25.744318 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:25.744289    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:11:25.815944 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:25.815907    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:11:25.849384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:25.836077    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:11:26.393865 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:26.393835    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:11:26.690219 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:26.690127    4301 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 31 23:11:26.690525 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:26.690484    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm" podUID="de434cd6-3610-47ac-ae7e-78e1d05cb0c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 31 23:11:26.751975 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:26.751534    4301 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 31 23:11:26.751975 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:26.751593    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722" podUID="ff5f8cb0-0541-4dae-9a86-8c3c0fdda126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 31 23:11:27.685951 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:27.685906    4301 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 31 23:11:27.686466 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:27.685988    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm" podUID="de434cd6-3610-47ac-ae7e-78e1d05cb0c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 31 23:11:27.744888 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:27.744853    4301 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 31 23:11:27.745163 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:27.745133    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722" podUID="ff5f8cb0-0541-4dae-9a86-8c3c0fdda126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 31 23:11:28.685312 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:28.685276    4301 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 31 23:11:28.685538 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:28.685340    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm" podUID="de434cd6-3610-47ac-ae7e-78e1d05cb0c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 31 23:11:28.747382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:28.745714    4301 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 31 23:11:28.747382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:28.745773    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722" podUID="ff5f8cb0-0541-4dae-9a86-8c3c0fdda126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 31 23:11:29.686378 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:29.685590    4301 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 31 23:11:29.686378 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:29.685678    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm" podUID="de434cd6-3610-47ac-ae7e-78e1d05cb0c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 31 23:11:29.745790 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:29.745752    4301 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 31 23:11:29.746078 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:29.746046    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722" podUID="ff5f8cb0-0541-4dae-9a86-8c3c0fdda126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 31 23:11:30.686373 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:30.685835    4301 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 31 23:11:30.686373 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:30.685908    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm" podUID="de434cd6-3610-47ac-ae7e-78e1d05cb0c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 31 23:11:30.749380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:30.748528    4301 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 31 23:11:30.749380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:30.748607    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722" podUID="ff5f8cb0-0541-4dae-9a86-8c3c0fdda126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 31 23:11:31.742065 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:31.742026    4301 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 31 23:11:31.742580 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:31.742108    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722" podUID="ff5f8cb0-0541-4dae-9a86-8c3c0fdda126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 31 23:11:32.098437 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:32.097387    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:11:32.098437 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:32.098316    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:11:32.102848 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:32.102821    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:11:32.546737 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:11:32.546704    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:11:32.747133 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:32.747103    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:11:32.747627 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:32.747468    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:11:32.748292 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:32.748270    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:11:35.997964 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:35.997932    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:11:39.733537 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.730322    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"]
Oct 31 23:11:39.781107 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.778653    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"]
Oct 31 23:11:39.781107 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.778725    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"]
Oct 31 23:11:39.781107 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.780466    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:11:39.784003 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.783948    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"]
Oct 31 23:11:39.784142 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.784023    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"]
Oct 31 23:11:39.784142 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.784055    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:11:39.788757 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.788730    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:11:39.793277 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.793252    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"]
Oct 31 23:11:39.861653 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.861612    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-workload-certs\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:11:39.861820 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.861675    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13c9f33c-af79-4938-b54f-142dc2f72218-istio-token\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:11:39.861820 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.861742    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-istio-envoy\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:11:39.861820 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.861771    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-workload-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:11:39.861820 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.861788    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-istio-envoy\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:11:39.861820 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.861813    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b097bb-1a53-48cc-8499-f13014278d10-istio-podinfo\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:11:39.862038 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.861872    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-workload-certs\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:11:39.862038 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.861911    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2abed568-6cc8-4336-b427-b8a976686c3f-istio-podinfo\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:11:39.862038 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.861928    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-istio-data\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:11:39.862038 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.861945    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lggv\" (UniqueName: \"kubernetes.io/projected/f7b097bb-1a53-48cc-8499-f13014278d10-kube-api-access-6lggv\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:11:39.862038 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.861962    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-workload-certs\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:11:39.862286 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.862101    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2abed568-6cc8-4336-b427-b8a976686c3f-istio-token\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:11:39.862286 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.862166    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-workload-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:11:39.862286 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.862197    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-credential-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:11:39.862286 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.862243    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvdtp\" (UniqueName: \"kubernetes.io/projected/2abed568-6cc8-4336-b427-b8a976686c3f-kube-api-access-rvdtp\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:11:39.862474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.862347    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b097bb-1a53-48cc-8499-f13014278d10-istiod-ca-cert\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:11:39.862474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.862420    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-credential-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:11:39.862474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.862451    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7b097bb-1a53-48cc-8499-f13014278d10-istio-token\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:11:39.862591 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.862480    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-istio-data\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:11:39.862591 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.862504    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2abed568-6cc8-4336-b427-b8a976686c3f-istiod-ca-cert\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:11:39.862591 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.862538    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-credential-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:11:39.862591 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.862568    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-workload-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:11:39.862717 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.862607    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfd2t\" (UniqueName: \"kubernetes.io/projected/13c9f33c-af79-4938-b54f-142dc2f72218-kube-api-access-nfd2t\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:11:39.862717 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.862623    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-istio-data\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:11:39.862717 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.862683    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13c9f33c-af79-4938-b54f-142dc2f72218-istio-podinfo\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:11:39.862807 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.862722    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13c9f33c-af79-4938-b54f-142dc2f72218-istiod-ca-cert\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:11:39.862807 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.862797    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-istio-envoy\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:11:39.963627 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.963588    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfd2t\" (UniqueName: \"kubernetes.io/projected/13c9f33c-af79-4938-b54f-142dc2f72218-kube-api-access-nfd2t\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:11:39.963627 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.963631    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-istio-data\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:11:39.963876 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.963658    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13c9f33c-af79-4938-b54f-142dc2f72218-istio-podinfo\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:11:39.963876 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.963680    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13c9f33c-af79-4938-b54f-142dc2f72218-istiod-ca-cert\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:11:39.963876 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.963710    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-istio-envoy\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:11:39.963876 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.963748    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-workload-certs\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:11:39.963876 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.963771    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13c9f33c-af79-4938-b54f-142dc2f72218-istio-token\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:11:39.963876 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.963799    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-istio-envoy\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:11:39.963876 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.963825    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-workload-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:11:39.963876 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.963849    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-istio-envoy\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:11:39.963876 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.963873    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b097bb-1a53-48cc-8499-f13014278d10-istio-podinfo\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:11:39.964314 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.963905    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-workload-certs\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:11:39.964314 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.963933    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2abed568-6cc8-4336-b427-b8a976686c3f-istio-podinfo\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:11:39.964314 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.963962    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-istio-data\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:11:39.964314 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.963992    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lggv\" (UniqueName: \"kubernetes.io/projected/f7b097bb-1a53-48cc-8499-f13014278d10-kube-api-access-6lggv\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:11:39.964314 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.964023    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-workload-certs\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:11:39.964314 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.964099    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-istio-data\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:11:39.964314 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.964102    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2abed568-6cc8-4336-b427-b8a976686c3f-istio-token\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:11:39.964314 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.964139    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-workload-certs\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:11:39.964314 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.964183    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-workload-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:11:39.964314 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.964219    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-credential-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:11:39.964314 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.964276    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvdtp\" (UniqueName: \"kubernetes.io/projected/2abed568-6cc8-4336-b427-b8a976686c3f-kube-api-access-rvdtp\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:11:39.964314 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.964290    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-workload-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:11:39.965038 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.964349    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b097bb-1a53-48cc-8499-f13014278d10-istiod-ca-cert\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:11:39.965038 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.964571    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13c9f33c-af79-4938-b54f-142dc2f72218-istiod-ca-cert\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:11:39.965038 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.964738    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-workload-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:11:39.965038 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.964768    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-istio-data\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:11:39.965385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.965329    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-credential-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:11:39.965570 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.965548    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-workload-certs\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:11:39.965691 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.965668    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-credential-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:11:39.965776 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.965759    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7b097bb-1a53-48cc-8499-f13014278d10-istio-token\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:11:39.965845 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.965643    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b097bb-1a53-48cc-8499-f13014278d10-istiod-ca-cert\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:11:39.966036 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.966015    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-istio-data\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:11:39.966118 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.966052    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2abed568-6cc8-4336-b427-b8a976686c3f-istiod-ca-cert\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:11:39.966118 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.966098    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-credential-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:11:39.966227 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.966122    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-workload-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:11:39.966530 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.966507    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-workload-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:11:39.966777 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.966750    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-istio-data\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:11:39.966988 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.966964    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-istio-envoy\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:11:39.966988 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.966974    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-istio-envoy\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:11:39.967122 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.966964    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-credential-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:11:39.967122 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.967082    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-credential-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:11:39.967230 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.967182    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2abed568-6cc8-4336-b427-b8a976686c3f-istiod-ca-cert\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:11:39.967272 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.967249    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-workload-certs\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:11:39.967468 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.967448    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13c9f33c-af79-4938-b54f-142dc2f72218-istio-podinfo\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:11:39.967519 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.967445    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b097bb-1a53-48cc-8499-f13014278d10-istio-podinfo\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:11:39.968071 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.968051    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2abed568-6cc8-4336-b427-b8a976686c3f-istio-podinfo\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:11:39.968207 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.968183    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-istio-envoy\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:11:39.976522 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.976470    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2abed568-6cc8-4336-b427-b8a976686c3f-istio-token\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:11:39.976725 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.976701    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13c9f33c-af79-4938-b54f-142dc2f72218-istio-token\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:11:39.977152 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.977126    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfd2t\" (UniqueName: \"kubernetes.io/projected/13c9f33c-af79-4938-b54f-142dc2f72218-kube-api-access-nfd2t\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:11:39.977338 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.977319    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvdtp\" (UniqueName: \"kubernetes.io/projected/2abed568-6cc8-4336-b427-b8a976686c3f-kube-api-access-rvdtp\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:11:39.977699 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.977684    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7b097bb-1a53-48cc-8499-f13014278d10-istio-token\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:11:39.977913 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:39.977897    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lggv\" (UniqueName: \"kubernetes.io/projected/f7b097bb-1a53-48cc-8499-f13014278d10-kube-api-access-6lggv\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:11:40.099410 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:40.099312    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:11:40.110202 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:40.110172    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:11:40.119476 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:40.119450    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:11:40.399568 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:40.399440    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"]
Oct 31 23:11:40.401863 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:11:40.401838    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2abed568_6cc8_4336_b427_b8a976686c3f.slice/crio-dbc5dee8664533599285b517eb4f4566ceaf682edd53a9b2d8f32c4fe20e6bb8 WatchSource:0}: Error finding container dbc5dee8664533599285b517eb4f4566ceaf682edd53a9b2d8f32c4fe20e6bb8: Status 404 returned error can't find the container with id dbc5dee8664533599285b517eb4f4566ceaf682edd53a9b2d8f32c4fe20e6bb8
Oct 31 23:11:40.404619 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:40.404584    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:11:40.404726 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:40.404658    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:11:40.404726 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:40.404695    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:11:40.445768 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:40.435603    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"]
Oct 31 23:11:40.457407 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:40.446528    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"]
Oct 31 23:11:40.462139 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:11:40.459051    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b097bb_1a53_48cc_8499_f13014278d10.slice/crio-b6974c19bb26558dbe0cb42c2c09ef9c7bcd108cd2344967a875996e853b223a WatchSource:0}: Error finding container b6974c19bb26558dbe0cb42c2c09ef9c7bcd108cd2344967a875996e853b223a: Status 404 returned error can't find the container with id b6974c19bb26558dbe0cb42c2c09ef9c7bcd108cd2344967a875996e853b223a
Oct 31 23:11:40.472790 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:40.466519    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:11:40.472790 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:40.466612    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:11:40.472790 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:40.466663    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:11:40.472790 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:11:40.472238    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13c9f33c_af79_4938_b54f_142dc2f72218.slice/crio-a211e6d6768d35751c97187f528ed99230b55b66032c59552e40e0586d414756 WatchSource:0}: Error finding container a211e6d6768d35751c97187f528ed99230b55b66032c59552e40e0586d414756: Status 404 returned error can't find the container with id a211e6d6768d35751c97187f528ed99230b55b66032c59552e40e0586d414756
Oct 31 23:11:40.487158 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:40.486291    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:11:40.487158 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:40.486411    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:11:40.487158 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:40.486454    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:11:40.510420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:40.510391    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw" event={"ID":"2abed568-6cc8-4336-b427-b8a976686c3f","Type":"ContainerStarted","Data":"dbc5dee8664533599285b517eb4f4566ceaf682edd53a9b2d8f32c4fe20e6bb8"}
Oct 31 23:11:40.511665 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:40.511640    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz" event={"ID":"13c9f33c-af79-4938-b54f-142dc2f72218","Type":"ContainerStarted","Data":"a211e6d6768d35751c97187f528ed99230b55b66032c59552e40e0586d414756"}
Oct 31 23:11:40.512713 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:40.512695    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9" event={"ID":"f7b097bb-1a53-48cc-8499-f13014278d10","Type":"ContainerStarted","Data":"b6974c19bb26558dbe0cb42c2c09ef9c7bcd108cd2344967a875996e853b223a"}
Oct 31 23:11:41.520474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:41.520439    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz" event={"ID":"13c9f33c-af79-4938-b54f-142dc2f72218","Type":"ContainerStarted","Data":"1a4188deb86605e67944e8210642d0dd1d1d233c923ccf29c897f6a81d5f1066"}
Oct 31 23:11:41.522564 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:41.522536    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9" event={"ID":"f7b097bb-1a53-48cc-8499-f13014278d10","Type":"ContainerStarted","Data":"0b1a39944e5d8230624e6db233fd50248f84730c3c5b5c9ded424d0a432d221f"}
Oct 31 23:11:41.524346 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:41.524325    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw" event={"ID":"2abed568-6cc8-4336-b427-b8a976686c3f","Type":"ContainerStarted","Data":"23ab4029c0559ed24c259d479471c886a4ee16cfe31e79979cf99e43f8cf7bdc"}
Oct 31 23:11:41.554917 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:41.554857    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz" podStartSLOduration=2.55483925 podStartE2EDuration="2.55483925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:11:41.552872577 +0000 UTC m=+1320.300062096" watchObservedRunningTime="2026-10-31 23:11:41.55483925 +0000 UTC m=+1320.302028756"
Oct 31 23:11:41.649378 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:41.641901    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9" podStartSLOduration=2.6418799120000003 podStartE2EDuration="2.641879912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:11:41.6135195 +0000 UTC m=+1320.360709004" watchObservedRunningTime="2026-10-31 23:11:41.641879912 +0000 UTC m=+1320.389069421"
Oct 31 23:11:42.101820 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:42.101790    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:11:42.102155 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:42.102136    4301 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 31 23:11:42.102289 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:42.102269    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw" podUID="2abed568-6cc8-4336-b427-b8a976686c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 31 23:11:42.115391 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:42.114826    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:11:42.116952 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:42.116919    4301 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 31 23:11:42.117163 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:42.117134    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9" podUID="f7b097bb-1a53-48cc-8499-f13014278d10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 31 23:11:42.120318 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:42.120304    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:11:42.121523 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:42.121502    4301 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 31 23:11:42.121701 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:42.121682    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz" podUID="13c9f33c-af79-4938-b54f-142dc2f72218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 31 23:11:43.099786 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:43.099748    4301 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 31 23:11:43.100396 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:43.100339    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw" podUID="2abed568-6cc8-4336-b427-b8a976686c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 31 23:11:43.118239 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:43.118205    4301 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 31 23:11:43.118511 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:43.118482    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9" podUID="f7b097bb-1a53-48cc-8499-f13014278d10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 31 23:11:43.136057 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:43.135933    4301 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 31 23:11:43.136057 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:43.135985    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz" podUID="13c9f33c-af79-4938-b54f-142dc2f72218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 31 23:11:44.100783 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:44.100744    4301 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 31 23:11:44.101405 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:44.101372    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw" podUID="2abed568-6cc8-4336-b427-b8a976686c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 31 23:11:44.113883 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:44.113767    4301 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 31 23:11:44.113883 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:44.113824    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9" podUID="f7b097bb-1a53-48cc-8499-f13014278d10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 31 23:11:44.124553 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:44.124521    4301 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 31 23:11:44.124792 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:44.124765    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz" podUID="13c9f33c-af79-4938-b54f-142dc2f72218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 31 23:11:45.101833 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:45.101344    4301 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 31 23:11:45.101833 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:45.101440    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw" podUID="2abed568-6cc8-4336-b427-b8a976686c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 31 23:11:45.110765 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:45.110656    4301 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 31 23:11:45.110765 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:45.110708    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9" podUID="f7b097bb-1a53-48cc-8499-f13014278d10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 31 23:11:45.122181 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:45.121683    4301 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 31 23:11:45.122181 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:45.121734    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz" podUID="13c9f33c-af79-4938-b54f-142dc2f72218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 31 23:11:46.104126 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:46.104073    4301 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 31 23:11:46.104849 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:46.104150    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw" podUID="2abed568-6cc8-4336-b427-b8a976686c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 31 23:11:46.111148 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:46.111119    4301 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 31 23:11:46.111287 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:46.111174    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9" podUID="f7b097bb-1a53-48cc-8499-f13014278d10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 31 23:11:46.120249 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:46.120224    4301 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 31 23:11:46.120376 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:46.120263    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz" podUID="13c9f33c-af79-4938-b54f-142dc2f72218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 31 23:11:47.102503 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:47.102468    4301 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 31 23:11:47.103546 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:47.103017    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw" podUID="2abed568-6cc8-4336-b427-b8a976686c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 31 23:11:47.111418 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:47.111297    4301 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 31 23:11:47.111418 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:47.111345    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9" podUID="f7b097bb-1a53-48cc-8499-f13014278d10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 31 23:11:47.120656 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:47.120578    4301 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 31 23:11:47.120656 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:47.120614    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz" podUID="13c9f33c-af79-4938-b54f-142dc2f72218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 31 23:11:48.182713 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:48.182677    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:11:48.192498 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:48.192468    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:11:48.193848 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:48.193797    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:11:48.261198 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:48.261099    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw" podStartSLOduration=9.261080807999999 podStartE2EDuration="9.261080808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:11:41.650783007 +0000 UTC m=+1320.397972512" watchObservedRunningTime="2026-10-31 23:11:48.261080808 +0000 UTC m=+1327.008270317"
Oct 31 23:11:48.429809 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:48.429780    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:11:48.430627 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:48.430608    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:11:48.431396 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:48.431346    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:11:48.524545 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:48.524465    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:11:48.525147 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:48.525124    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:11:48.526602 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:48.526581    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:11:54.160044 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:54.160010    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"]
Oct 31 23:11:54.166536 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:54.166507    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:11:54.175540 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:54.175514    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-67fce9\"/\"distinct-2-w3-67fce9-kserve-self-signed-certs\""
Oct 31 23:11:54.176524 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:54.176495    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"]
Oct 31 23:11:54.254057 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:54.254023    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9a165f7-116b-4c15-9836-349bebeab135-dshm\") pod \"distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc\" (UID: \"d9a165f7-116b-4c15-9836-349bebeab135\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:11:54.254257 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:54.254071    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvhks\" (UniqueName: \"kubernetes.io/projected/d9a165f7-116b-4c15-9836-349bebeab135-kube-api-access-wvhks\") pod \"distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc\" (UID: \"d9a165f7-116b-4c15-9836-349bebeab135\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:11:54.254257 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:54.254101    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9a165f7-116b-4c15-9836-349bebeab135-tmp-dir\") pod \"distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc\" (UID: \"d9a165f7-116b-4c15-9836-349bebeab135\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:11:54.254422 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:54.254251    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9a165f7-116b-4c15-9836-349bebeab135-home\") pod \"distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc\" (UID: \"d9a165f7-116b-4c15-9836-349bebeab135\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:11:54.254422 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:54.254390    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9a165f7-116b-4c15-9836-349bebeab135-tls-certs\") pod \"distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc\" (UID: \"d9a165f7-116b-4c15-9836-349bebeab135\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:11:54.254529 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:54.254508    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9a165f7-116b-4c15-9836-349bebeab135-model-cache\") pod \"distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc\" (UID: \"d9a165f7-116b-4c15-9836-349bebeab135\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:11:54.374383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:54.374252    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9a165f7-116b-4c15-9836-349bebeab135-tls-certs\") pod \"distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc\" (UID: \"d9a165f7-116b-4c15-9836-349bebeab135\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:11:54.374941 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:54.374388    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9a165f7-116b-4c15-9836-349bebeab135-model-cache\") pod \"distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc\" (UID: \"d9a165f7-116b-4c15-9836-349bebeab135\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:11:54.374941 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:54.374488    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9a165f7-116b-4c15-9836-349bebeab135-dshm\") pod \"distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc\" (UID: \"d9a165f7-116b-4c15-9836-349bebeab135\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:11:54.374941 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:54.374530    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvhks\" (UniqueName: \"kubernetes.io/projected/d9a165f7-116b-4c15-9836-349bebeab135-kube-api-access-wvhks\") pod \"distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc\" (UID: \"d9a165f7-116b-4c15-9836-349bebeab135\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:11:54.374941 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:54.374563    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9a165f7-116b-4c15-9836-349bebeab135-tmp-dir\") pod \"distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc\" (UID: \"d9a165f7-116b-4c15-9836-349bebeab135\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:11:54.374941 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:54.374596    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9a165f7-116b-4c15-9836-349bebeab135-home\") pod \"distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc\" (UID: \"d9a165f7-116b-4c15-9836-349bebeab135\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:11:54.375231 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:54.375093    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9a165f7-116b-4c15-9836-349bebeab135-home\") pod \"distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc\" (UID: \"d9a165f7-116b-4c15-9836-349bebeab135\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:11:54.394388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:54.378107    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9a165f7-116b-4c15-9836-349bebeab135-model-cache\") pod \"distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc\" (UID: \"d9a165f7-116b-4c15-9836-349bebeab135\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:11:54.394388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:54.378374    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9a165f7-116b-4c15-9836-349bebeab135-tmp-dir\") pod \"distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc\" (UID: \"d9a165f7-116b-4c15-9836-349bebeab135\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:11:54.432172 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:54.432047    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9a165f7-116b-4c15-9836-349bebeab135-tls-certs\") pod \"distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc\" (UID: \"d9a165f7-116b-4c15-9836-349bebeab135\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:11:54.445977 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:54.445940    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9a165f7-116b-4c15-9836-349bebeab135-dshm\") pod \"distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc\" (UID: \"d9a165f7-116b-4c15-9836-349bebeab135\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:11:54.471391 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:54.452718    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvhks\" (UniqueName: \"kubernetes.io/projected/d9a165f7-116b-4c15-9836-349bebeab135-kube-api-access-wvhks\") pod \"distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc\" (UID: \"d9a165f7-116b-4c15-9836-349bebeab135\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:11:54.495379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:54.489587    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:11:54.794034 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:54.794010    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"]
Oct 31 23:11:54.796050 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:11:54.796022    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9a165f7_116b_4c15_9836_349bebeab135.slice/crio-c2ab3e6b2fabcd22b0a0d057ca767480fe53eafbc29201a53242351a2809a0e0 WatchSource:0}: Error finding container c2ab3e6b2fabcd22b0a0d057ca767480fe53eafbc29201a53242351a2809a0e0: Status 404 returned error can't find the container with id c2ab3e6b2fabcd22b0a0d057ca767480fe53eafbc29201a53242351a2809a0e0
Oct 31 23:11:55.709299 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:55.709263    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" event={"ID":"d9a165f7-116b-4c15-9836-349bebeab135","Type":"ContainerStarted","Data":"04ed85ed4e4daae0bc0bd6980041d7768a9d0d8b4a00a987e38724cc694e38fb"}
Oct 31 23:11:55.709724 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:55.709310    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" event={"ID":"d9a165f7-116b-4c15-9836-349bebeab135","Type":"ContainerStarted","Data":"c2ab3e6b2fabcd22b0a0d057ca767480fe53eafbc29201a53242351a2809a0e0"}
Oct 31 23:11:55.727043 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:55.726983    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podStartSLOduration=1.726964102 podStartE2EDuration="1.726964102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:11:55.726573489 +0000 UTC m=+1334.473763007" watchObservedRunningTime="2026-10-31 23:11:55.726964102 +0000 UTC m=+1334.474153609"
Oct 31 23:11:56.490481 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:11:56.490448    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:12:00.581385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:00.579437    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:01.069384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:01.067461    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:01.668519 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:01.668468    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:01.778438 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:01.778393    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:02.341689 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:02.341646    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 23:12:02.386682 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:02.386636    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:02.542396 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:02.539460    4301 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 23:12:02.542396 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:02.539527    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn" podUID="84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:02.746451 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:12:02.739628    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:12:04.490546 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:04.490513    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:12:04.504513 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:04.504483    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:12:05.091610 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:05.091579    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:12:05.987059 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:05.987012    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"]
Oct 31 23:12:05.990666 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:05.990644    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:12:06.002887 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:06.002812    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7d7eed\"/\"default-dockercfg-lql6f\""
Oct 31 23:12:06.002887 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:06.002825    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7d7eed\"/\"distinct-2-w6-7d7eed-kserve-self-signed-certs\""
Oct 31 23:12:06.003650 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:06.003629    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-7d7eed\"/\"openshift-service-ca.crt\""
Oct 31 23:12:06.003848 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:06.003656    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-7d7eed\"/\"kube-root-ca.crt\""
Oct 31 23:12:06.004912 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:06.004890    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"]
Oct 31 23:12:06.094099 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:06.094064    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2a4d961-adc7-4f38-9825-33f1230ddadb-tmp-dir\") pod \"distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8\" (UID: \"f2a4d961-adc7-4f38-9825-33f1230ddadb\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:12:06.094294 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:06.094194    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlxsj\" (UniqueName: \"kubernetes.io/projected/f2a4d961-adc7-4f38-9825-33f1230ddadb-kube-api-access-dlxsj\") pod \"distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8\" (UID: \"f2a4d961-adc7-4f38-9825-33f1230ddadb\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:12:06.094366 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:06.094293    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2a4d961-adc7-4f38-9825-33f1230ddadb-tls-certs\") pod \"distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8\" (UID: \"f2a4d961-adc7-4f38-9825-33f1230ddadb\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:12:06.094428 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:06.094378    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2a4d961-adc7-4f38-9825-33f1230ddadb-dshm\") pod \"distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8\" (UID: \"f2a4d961-adc7-4f38-9825-33f1230ddadb\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:12:06.094481 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:06.094445    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2a4d961-adc7-4f38-9825-33f1230ddadb-home\") pod \"distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8\" (UID: \"f2a4d961-adc7-4f38-9825-33f1230ddadb\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:12:06.094532 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:06.094508    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2a4d961-adc7-4f38-9825-33f1230ddadb-model-cache\") pod \"distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8\" (UID: \"f2a4d961-adc7-4f38-9825-33f1230ddadb\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:12:06.196926 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:06.196887    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2a4d961-adc7-4f38-9825-33f1230ddadb-tmp-dir\") pod \"distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8\" (UID: \"f2a4d961-adc7-4f38-9825-33f1230ddadb\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:12:06.198930 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:06.198904    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlxsj\" (UniqueName: \"kubernetes.io/projected/f2a4d961-adc7-4f38-9825-33f1230ddadb-kube-api-access-dlxsj\") pod \"distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8\" (UID: \"f2a4d961-adc7-4f38-9825-33f1230ddadb\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:12:06.199852 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:06.199828    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2a4d961-adc7-4f38-9825-33f1230ddadb-tls-certs\") pod \"distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8\" (UID: \"f2a4d961-adc7-4f38-9825-33f1230ddadb\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:12:06.200566 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:06.200542    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2a4d961-adc7-4f38-9825-33f1230ddadb-dshm\") pod \"distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8\" (UID: \"f2a4d961-adc7-4f38-9825-33f1230ddadb\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:12:06.201228 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:06.201210    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2a4d961-adc7-4f38-9825-33f1230ddadb-home\") pod \"distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8\" (UID: \"f2a4d961-adc7-4f38-9825-33f1230ddadb\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:12:06.210559 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:06.210536    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2a4d961-adc7-4f38-9825-33f1230ddadb-model-cache\") pod \"distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8\" (UID: \"f2a4d961-adc7-4f38-9825-33f1230ddadb\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:12:06.211141 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:06.210425    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2a4d961-adc7-4f38-9825-33f1230ddadb-home\") pod \"distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8\" (UID: \"f2a4d961-adc7-4f38-9825-33f1230ddadb\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:12:06.211279 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:06.198787    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2a4d961-adc7-4f38-9825-33f1230ddadb-tmp-dir\") pod \"distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8\" (UID: \"f2a4d961-adc7-4f38-9825-33f1230ddadb\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:12:06.211394 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:06.210985    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2a4d961-adc7-4f38-9825-33f1230ddadb-model-cache\") pod \"distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8\" (UID: \"f2a4d961-adc7-4f38-9825-33f1230ddadb\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:12:06.226382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:06.223136    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2a4d961-adc7-4f38-9825-33f1230ddadb-tls-certs\") pod \"distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8\" (UID: \"f2a4d961-adc7-4f38-9825-33f1230ddadb\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:12:06.230675 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:06.230644    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2a4d961-adc7-4f38-9825-33f1230ddadb-dshm\") pod \"distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8\" (UID: \"f2a4d961-adc7-4f38-9825-33f1230ddadb\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:12:06.254232 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:06.254066    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlxsj\" (UniqueName: \"kubernetes.io/projected/f2a4d961-adc7-4f38-9825-33f1230ddadb-kube-api-access-dlxsj\") pod \"distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8\" (UID: \"f2a4d961-adc7-4f38-9825-33f1230ddadb\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:12:06.306872 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:06.306836    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:12:06.694555 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:06.694518    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"]
Oct 31 23:12:06.705700 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:12:06.705664    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2a4d961_adc7_4f38_9825_33f1230ddadb.slice/crio-de04a4f4bd0a3fdccdf7e42c3c282b2189c5f0419d425af7871cec44a58fa2d6 WatchSource:0}: Error finding container de04a4f4bd0a3fdccdf7e42c3c282b2189c5f0419d425af7871cec44a58fa2d6: Status 404 returned error can't find the container with id de04a4f4bd0a3fdccdf7e42c3c282b2189c5f0419d425af7871cec44a58fa2d6
Oct 31 23:12:07.094421 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:07.094318    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" event={"ID":"f2a4d961-adc7-4f38-9825-33f1230ddadb","Type":"ContainerStarted","Data":"5890f9d31e51eda37394c7bcc3dc59d847ab904af5ae24c0c9a4bf14aa6a5cc2"}
Oct 31 23:12:07.094421 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:07.094400    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" event={"ID":"f2a4d961-adc7-4f38-9825-33f1230ddadb","Type":"ContainerStarted","Data":"de04a4f4bd0a3fdccdf7e42c3c282b2189c5f0419d425af7871cec44a58fa2d6"}
Oct 31 23:12:07.129379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:07.122744    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podStartSLOduration=2.122723093 podStartE2EDuration="2.122723093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:12:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:12:07.121558654 +0000 UTC m=+1345.868748161" watchObservedRunningTime="2026-10-31 23:12:07.122723093 +0000 UTC m=+1345.869912600"
Oct 31 23:12:07.308524 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:07.308492    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:12:09.178281 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.176718    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"]
Oct 31 23:12:09.211245 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.211213    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"]
Oct 31 23:12:09.211601 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.211574    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:12:09.220711 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.218518    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-135248\"/\"distinct-w4-135248-kserve-self-signed-certs\""
Oct 31 23:12:09.254198 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.254165    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca870469-c464-4272-a7d7-61081a3d2afc-home\") pod \"distinct-w4-135248-kserve-755d98c8b6-4zn56\" (UID: \"ca870469-c464-4272-a7d7-61081a3d2afc\") " pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:12:09.254406 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.254215    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca870469-c464-4272-a7d7-61081a3d2afc-tmp-dir\") pod \"distinct-w4-135248-kserve-755d98c8b6-4zn56\" (UID: \"ca870469-c464-4272-a7d7-61081a3d2afc\") " pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:12:09.254406 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.254368    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca870469-c464-4272-a7d7-61081a3d2afc-tls-certs\") pod \"distinct-w4-135248-kserve-755d98c8b6-4zn56\" (UID: \"ca870469-c464-4272-a7d7-61081a3d2afc\") " pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:12:09.254531 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.254450    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vx7b\" (UniqueName: \"kubernetes.io/projected/ca870469-c464-4272-a7d7-61081a3d2afc-kube-api-access-8vx7b\") pod \"distinct-w4-135248-kserve-755d98c8b6-4zn56\" (UID: \"ca870469-c464-4272-a7d7-61081a3d2afc\") " pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:12:09.254531 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.254510    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca870469-c464-4272-a7d7-61081a3d2afc-dshm\") pod \"distinct-w4-135248-kserve-755d98c8b6-4zn56\" (UID: \"ca870469-c464-4272-a7d7-61081a3d2afc\") " pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:12:09.254636 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.254547    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca870469-c464-4272-a7d7-61081a3d2afc-model-cache\") pod \"distinct-w4-135248-kserve-755d98c8b6-4zn56\" (UID: \"ca870469-c464-4272-a7d7-61081a3d2afc\") " pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:12:09.282573 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.282542    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"]
Oct 31 23:12:09.294167 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.294095    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:12:09.301195 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.301164    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"]
Oct 31 23:12:09.365888 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.360422    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-istio-envoy\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:12:09.365888 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.360484    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/929c3031-62b4-4587-9f53-75de251130c8-istio-podinfo\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:12:09.365888 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.360529    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca870469-c464-4272-a7d7-61081a3d2afc-dshm\") pod \"distinct-w4-135248-kserve-755d98c8b6-4zn56\" (UID: \"ca870469-c464-4272-a7d7-61081a3d2afc\") " pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:12:09.376201 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.376155    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-credential-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:12:09.377986 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.377955    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-workload-certs\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:12:09.378211 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.378195    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-istio-data\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:12:09.378340 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.378326    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca870469-c464-4272-a7d7-61081a3d2afc-model-cache\") pod \"distinct-w4-135248-kserve-755d98c8b6-4zn56\" (UID: \"ca870469-c464-4272-a7d7-61081a3d2afc\") " pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:12:09.378475 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.378459    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc424\" (UniqueName: \"kubernetes.io/projected/929c3031-62b4-4587-9f53-75de251130c8-kube-api-access-mc424\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:12:09.378743 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.378709    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/929c3031-62b4-4587-9f53-75de251130c8-istiod-ca-cert\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:12:09.378888 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.378875    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca870469-c464-4272-a7d7-61081a3d2afc-home\") pod \"distinct-w4-135248-kserve-755d98c8b6-4zn56\" (UID: \"ca870469-c464-4272-a7d7-61081a3d2afc\") " pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:12:09.379002 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.378989    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca870469-c464-4272-a7d7-61081a3d2afc-tmp-dir\") pod \"distinct-w4-135248-kserve-755d98c8b6-4zn56\" (UID: \"ca870469-c464-4272-a7d7-61081a3d2afc\") " pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:12:09.379127 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.379113    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/929c3031-62b4-4587-9f53-75de251130c8-istio-token\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:12:09.379277 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.379263    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca870469-c464-4272-a7d7-61081a3d2afc-tls-certs\") pod \"distinct-w4-135248-kserve-755d98c8b6-4zn56\" (UID: \"ca870469-c464-4272-a7d7-61081a3d2afc\") " pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:12:09.379400 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.379385    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-workload-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:12:09.391403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.387474    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vx7b\" (UniqueName: \"kubernetes.io/projected/ca870469-c464-4272-a7d7-61081a3d2afc-kube-api-access-8vx7b\") pod \"distinct-w4-135248-kserve-755d98c8b6-4zn56\" (UID: \"ca870469-c464-4272-a7d7-61081a3d2afc\") " pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:12:09.391403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.388440    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca870469-c464-4272-a7d7-61081a3d2afc-model-cache\") pod \"distinct-w4-135248-kserve-755d98c8b6-4zn56\" (UID: \"ca870469-c464-4272-a7d7-61081a3d2afc\") " pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:12:09.412646 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.412610    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca870469-c464-4272-a7d7-61081a3d2afc-tmp-dir\") pod \"distinct-w4-135248-kserve-755d98c8b6-4zn56\" (UID: \"ca870469-c464-4272-a7d7-61081a3d2afc\") " pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:12:09.421490 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.421457    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca870469-c464-4272-a7d7-61081a3d2afc-home\") pod \"distinct-w4-135248-kserve-755d98c8b6-4zn56\" (UID: \"ca870469-c464-4272-a7d7-61081a3d2afc\") " pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:12:09.451644 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.451568    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca870469-c464-4272-a7d7-61081a3d2afc-tls-certs\") pod \"distinct-w4-135248-kserve-755d98c8b6-4zn56\" (UID: \"ca870469-c464-4272-a7d7-61081a3d2afc\") " pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:12:09.453527 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.453500    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca870469-c464-4272-a7d7-61081a3d2afc-dshm\") pod \"distinct-w4-135248-kserve-755d98c8b6-4zn56\" (UID: \"ca870469-c464-4272-a7d7-61081a3d2afc\") " pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:12:09.462216 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.462146    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vx7b\" (UniqueName: \"kubernetes.io/projected/ca870469-c464-4272-a7d7-61081a3d2afc-kube-api-access-8vx7b\") pod \"distinct-w4-135248-kserve-755d98c8b6-4zn56\" (UID: \"ca870469-c464-4272-a7d7-61081a3d2afc\") " pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:12:09.502182 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.502138    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-istio-envoy\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:12:09.502479 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.502457    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/929c3031-62b4-4587-9f53-75de251130c8-istio-podinfo\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:12:09.502626 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.502610    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-credential-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:12:09.515328 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.506391    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-workload-certs\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:12:09.515803 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.515768    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-istio-data\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:12:09.516534 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.516508    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc424\" (UniqueName: \"kubernetes.io/projected/929c3031-62b4-4587-9f53-75de251130c8-kube-api-access-mc424\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:12:09.529148 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.526612    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/929c3031-62b4-4587-9f53-75de251130c8-istiod-ca-cert\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:12:09.540817 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.540778    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/929c3031-62b4-4587-9f53-75de251130c8-istio-token\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:12:09.541139 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.541115    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-workload-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:12:09.557536 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.516219    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-istio-data\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:12:09.557750 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.540236    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/929c3031-62b4-4587-9f53-75de251130c8-istiod-ca-cert\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:12:09.558559 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.540471    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:12:09.559552 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.527549    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-workload-certs\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:12:09.559552 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.512193    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-credential-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:12:09.559552 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.547048    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-workload-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:12:09.559552 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.552794    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-istio-envoy\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:12:09.561212 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.561164    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/929c3031-62b4-4587-9f53-75de251130c8-istio-podinfo\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:12:09.583265 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.582755    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc424\" (UniqueName: \"kubernetes.io/projected/929c3031-62b4-4587-9f53-75de251130c8-kube-api-access-mc424\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:12:09.602548 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.602476    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/929c3031-62b4-4587-9f53-75de251130c8-istio-token\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:12:09.610846 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.610689    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:12:09.893719 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.893690    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"]
Oct 31 23:12:09.896077 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:12:09.896040    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca870469_c464_4272_a7d7_61081a3d2afc.slice/crio-74186068436c76f5f632ab84a4807374da41b81d49adc6c6f58094a4dc6948dc WatchSource:0}: Error finding container 74186068436c76f5f632ab84a4807374da41b81d49adc6c6f58094a4dc6948dc: Status 404 returned error can't find the container with id 74186068436c76f5f632ab84a4807374da41b81d49adc6c6f58094a4dc6948dc
Oct 31 23:12:09.953758 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.923580    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"]
Oct 31 23:12:09.990317 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.990261    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:12:09.990450 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.990412    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:12:09.990513 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:09.990467    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:12:10.128454 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:10.128417    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" event={"ID":"ca870469-c464-4272-a7d7-61081a3d2afc","Type":"ContainerStarted","Data":"59f03b727cae67bed70d755db3db359e263bf6397d33b55041a696e1a0ed47b5"}
Oct 31 23:12:10.128638 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:10.128466    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" event={"ID":"ca870469-c464-4272-a7d7-61081a3d2afc","Type":"ContainerStarted","Data":"74186068436c76f5f632ab84a4807374da41b81d49adc6c6f58094a4dc6948dc"}
Oct 31 23:12:10.130241 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:10.130210    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954" event={"ID":"929c3031-62b4-4587-9f53-75de251130c8","Type":"ContainerStarted","Data":"af1f952860c2f21a81872c77b77106eb28a8ef2e43e68dbdd858ba9bbb9d0b3f"}
Oct 31 23:12:10.130398 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:10.130260    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954" event={"ID":"929c3031-62b4-4587-9f53-75de251130c8","Type":"ContainerStarted","Data":"1f376b976a92e80502163d4a043c1cbd6b9d5306ffefaf56db2d307b2639099a"}
Oct 31 23:12:10.149205 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:10.149157    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podStartSLOduration=1.149139207 podStartE2EDuration="1.149139207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:12:10.147218113 +0000 UTC m=+1348.894407617" watchObservedRunningTime="2026-10-31 23:12:10.149139207 +0000 UTC m=+1348.896328720"
Oct 31 23:12:10.170579 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:10.170510    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954" podStartSLOduration=1.170492463 podStartE2EDuration="1.170492463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:12:10.168452795 +0000 UTC m=+1348.915642302" watchObservedRunningTime="2026-10-31 23:12:10.170492463 +0000 UTC m=+1348.917681970"
Oct 31 23:12:10.541559 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:10.541512    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:12:10.611128 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:10.611095    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:12:11.612949 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:11.612911    4301 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 23:12:11.613489 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:11.612963    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954" podUID="929c3031-62b4-4587-9f53-75de251130c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 23:12:12.611608 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:12.611570    4301 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 23:12:12.611784 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:12.611621    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954" podUID="929c3031-62b4-4587-9f53-75de251130c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 23:12:13.612238 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:13.612200    4301 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 23:12:13.612757 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:13.612256    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954" podUID="929c3031-62b4-4587-9f53-75de251130c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 23:12:14.611506 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:14.611470    4301 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 23:12:14.611685 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:14.611525    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954" podUID="929c3031-62b4-4587-9f53-75de251130c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 23:12:15.611706 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:15.611675    4301 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 23:12:15.612104 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:15.611725    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954" podUID="929c3031-62b4-4587-9f53-75de251130c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 23:12:16.317773 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:16.316843    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:12:16.331152 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:16.331122    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:12:16.612202 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:16.612113    4301 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 23:12:16.612645 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:16.612196    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954" podUID="929c3031-62b4-4587-9f53-75de251130c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 23:12:17.263762 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:17.263730    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:12:17.620439 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:17.620353    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:12:17.621533 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:17.621505    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:12:17.626471 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:17.626446    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:12:19.541305 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:19.541274    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:12:19.553939 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:19.553914    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:12:20.299540 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:20.299511    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:12:23.626790 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:23.626744    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:12:24.448385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.447454    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"]
Oct 31 23:12:24.455379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.454760    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:12:24.475386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.471506    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"]
Oct 31 23:12:24.612601 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.612550    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-istio-envoy\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:12:24.612601 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.612600    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94a46479-0793-4c0a-8d40-b342c343899c-istio-podinfo\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:12:24.612860 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.612624    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-istio-data\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:12:24.612860 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.612672    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-workload-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:12:24.612860 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.612701    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94a46479-0793-4c0a-8d40-b342c343899c-istiod-ca-cert\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:12:24.612860 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.612731    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-workload-certs\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:12:24.612860 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.612765    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-credential-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:12:24.612860 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.612790    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94a46479-0793-4c0a-8d40-b342c343899c-istio-token\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:12:24.612860 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.612845    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fv5f\" (UniqueName: \"kubernetes.io/projected/94a46479-0793-4c0a-8d40-b342c343899c-kube-api-access-6fv5f\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:12:24.661095 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.661064    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"]
Oct 31 23:12:24.715636 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.713745    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-istio-envoy\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:12:24.715636 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.713790    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94a46479-0793-4c0a-8d40-b342c343899c-istio-podinfo\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:12:24.715636 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.713816    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-istio-data\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:12:24.715636 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.713865    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-workload-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:12:24.715636 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.713892    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94a46479-0793-4c0a-8d40-b342c343899c-istiod-ca-cert\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:12:24.715636 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.713924    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-workload-certs\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:12:24.715636 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.713957    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-credential-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:12:24.715636 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.713981    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94a46479-0793-4c0a-8d40-b342c343899c-istio-token\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:12:24.715636 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.714032    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fv5f\" (UniqueName: \"kubernetes.io/projected/94a46479-0793-4c0a-8d40-b342c343899c-kube-api-access-6fv5f\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:12:24.715636 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.714839    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-workload-certs\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:12:24.715636 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.714855    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94a46479-0793-4c0a-8d40-b342c343899c-istiod-ca-cert\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:12:24.715636 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.715085    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-credential-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:12:24.715636 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.715553    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-workload-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:12:24.715636 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.715568    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-istio-data\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:12:24.723936 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.723906    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94a46479-0793-4c0a-8d40-b342c343899c-istio-podinfo\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:12:24.736181 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.735844    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94a46479-0793-4c0a-8d40-b342c343899c-istio-token\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:12:24.749092 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.749034    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-istio-envoy\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:12:24.758186 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.758131    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fv5f\" (UniqueName: \"kubernetes.io/projected/94a46479-0793-4c0a-8d40-b342c343899c-kube-api-access-6fv5f\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:12:24.758970 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.758946    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"]
Oct 31 23:12:24.759081 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.759067    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:12:24.821675 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.820233    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc182c32-7804-4e3f-b6eb-d22418055d75-istiod-ca-cert\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:12:24.821675 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.820282    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-workload-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:12:24.821675 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.820311    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc182c32-7804-4e3f-b6eb-d22418055d75-istio-token\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:12:24.821675 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.820474    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-workload-certs\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:12:24.821675 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.820515    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-istio-envoy\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:12:24.821675 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.820546    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc182c32-7804-4e3f-b6eb-d22418055d75-istio-podinfo\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:12:24.821675 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.820619    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-istio-data\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:12:24.821675 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.820650    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf4cg\" (UniqueName: \"kubernetes.io/projected/cc182c32-7804-4e3f-b6eb-d22418055d75-kube-api-access-pf4cg\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:12:24.821675 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.820693    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-credential-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:12:24.850637 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.850598    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"]
Oct 31 23:12:24.865051 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.865016    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:12:24.892012 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.891978    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"]
Oct 31 23:12:24.936831 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.936799    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-istio-data\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:12:24.940182 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.937607    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-istio-data\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:12:24.944600 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.944570    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf4cg\" (UniqueName: \"kubernetes.io/projected/cc182c32-7804-4e3f-b6eb-d22418055d75-kube-api-access-pf4cg\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:12:24.945142 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.945119    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istio-podinfo\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:12:24.945351 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.945333    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q72qt\" (UniqueName: \"kubernetes.io/projected/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-kube-api-access-q72qt\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:12:24.945516 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.945500    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-credential-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:12:24.945730 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.945717    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc182c32-7804-4e3f-b6eb-d22418055d75-istiod-ca-cert\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:12:24.948858 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.948823    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istiod-ca-cert\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:12:24.949136 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.949094    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-workload-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:12:24.949689 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.949662    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc182c32-7804-4e3f-b6eb-d22418055d75-istio-token\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:12:24.950259 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.950230    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-workload-certs\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:12:24.950763 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.948203    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc182c32-7804-4e3f-b6eb-d22418055d75-istiod-ca-cert\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:12:24.950907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.949622    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-workload-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:12:24.951268 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.951136    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istio-token\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:12:24.951488 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.951204    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-credential-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:12:24.951708 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.951691    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-workload-certs\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:12:24.952005 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.951987    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-workload-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:12:24.952761 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.952477    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:12:24.953300 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.953277    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istio-data\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:12:24.953684 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.953537    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-istio-envoy\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:12:25.005627 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.961628    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-credential-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:12:25.005627 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.975138    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istio-envoy\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:12:25.005627 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:24.975211    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc182c32-7804-4e3f-b6eb-d22418055d75-istio-podinfo\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:12:25.217425 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:25.030457    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-workload-certs\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:12:25.217425 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:25.070205    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc182c32-7804-4e3f-b6eb-d22418055d75-istio-podinfo\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:12:25.217425 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:25.084333    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istiod-ca-cert\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:12:25.217425 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:25.213904    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-workload-certs\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:12:25.217425 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:25.214034    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istio-token\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:12:25.217425 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:25.214160    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-workload-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:12:25.217425 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:25.214226    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istio-data\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:12:25.217425 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:25.214270    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-credential-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:12:25.217425 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:25.214296    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istio-envoy\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:12:25.217425 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:25.214483    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istio-podinfo\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:12:25.217425 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:25.214523    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q72qt\" (UniqueName: \"kubernetes.io/projected/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-kube-api-access-q72qt\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:12:25.217425 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:25.086003    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istiod-ca-cert\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:12:25.217425 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:25.189463    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf4cg\" (UniqueName: \"kubernetes.io/projected/cc182c32-7804-4e3f-b6eb-d22418055d75-kube-api-access-pf4cg\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:12:25.217425 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:25.205915    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc182c32-7804-4e3f-b6eb-d22418055d75-istio-token\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:12:25.217425 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:25.215920    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-workload-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:12:25.217425 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:25.216197    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istio-data\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:12:25.217425 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:25.216510    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-credential-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:12:25.218297 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:25.217436    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-workload-certs\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:12:25.241406 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:25.235863    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-istio-envoy\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:12:25.242574 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:25.242548    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istio-envoy\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:12:25.243805 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:25.243776    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istio-podinfo\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:12:25.259838 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:25.259202    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q72qt\" (UniqueName: \"kubernetes.io/projected/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-kube-api-access-q72qt\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:12:25.259990 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:25.259935    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istio-token\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:12:25.724037 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:25.718101    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:12:25.724037 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:25.723818    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:12:25.848975 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:25.848879    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"]
Oct 31 23:12:25.886776 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:25.886236    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:12:25.886776 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:25.886322    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:12:25.886776 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:25.886379    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:12:26.466400 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:26.447823    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 31 23:12:26.670974 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:26.670947    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"]
Oct 31 23:12:26.684424 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:26.684349    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln" event={"ID":"94a46479-0793-4c0a-8d40-b342c343899c","Type":"ContainerStarted","Data":"72c25d292a3c0c69df2a258f0a79ae1e0f3e877de33df70245d767b28e06262a"}
Oct 31 23:12:26.701544 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:26.701495    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:12:26.714602 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:26.714557    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:12:26.747520 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:26.716429    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:12:26.829290 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:26.745534    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:26.829290 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:26.768577    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:26.829290 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:26.794497    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:26.829290 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:26.827530    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:27.154473 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:27.152667    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"]
Oct 31 23:12:27.281384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:27.263284    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:12:27.281384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:27.263405    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:12:27.281384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:27.263447    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:12:27.334999 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:27.333792    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 23:12:27.458191 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:27.454879    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 31 23:12:27.458191 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:27.455333    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:12:27.661874 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:27.661823    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:27.908412 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:27.897179    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:28.047440 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:28.029289    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:12:28.223213 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:28.223174    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz" event={"ID":"3b0d47aa-36ab-46fa-b41f-20a71f2256cd","Type":"ContainerStarted","Data":"3f7a2fd73900e2cc79efa2b37dcf175a5a4f86068a67a5f4c2db63cc055207bb"}
Oct 31 23:12:28.258060 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:28.258018    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:28.276978 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:28.276917    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:28.297560 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:28.297459    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:28.334684 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:28.334653    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:12:28.358932 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:28.346314    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:28.358932 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:28.347044    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:12:28.368670 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:28.368648    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:12:28.405787 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:28.387421    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln" event={"ID":"94a46479-0793-4c0a-8d40-b342c343899c","Type":"ContainerStarted","Data":"06b0ca79d22ed7dba1e69c509321737c3f64bb6eac466ce2916510c61b88ec14"}
Oct 31 23:12:28.420772 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:28.420005    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:12:28.442680 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:28.442634    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq" event={"ID":"cc182c32-7804-4e3f-b6eb-d22418055d75","Type":"ContainerStarted","Data":"467f2796fba9a78feea970bfd89d5339722815972fa4f941313a98e5736b75a8"}
Oct 31 23:12:28.657831 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:28.657768    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln" podStartSLOduration=4.657744763 podStartE2EDuration="4.657744763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:12:28.518684578 +0000 UTC m=+1367.265874088" watchObservedRunningTime="2026-10-31 23:12:28.657744763 +0000 UTC m=+1367.404934269"
Oct 31 23:12:28.966440 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:28.966323    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:12:28.966893 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:28.966501    4301 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 23:12:28.966893 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:28.966546    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln" podUID="94a46479-0793-4c0a-8d40-b342c343899c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 23:12:29.467938 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:29.467902    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz" event={"ID":"3b0d47aa-36ab-46fa-b41f-20a71f2256cd","Type":"ContainerStarted","Data":"8b7f17202fa4b84b3448c8fc2b9dff61a7e2c111c38a19277ce96ba1bea1125d"}
Oct 31 23:12:29.470094 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:29.470065    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq" event={"ID":"cc182c32-7804-4e3f-b6eb-d22418055d75","Type":"ContainerStarted","Data":"347cbed007c562620ce4d4e70ee07825b288160e40b24cadcb7d7f352c487873"}
Oct 31 23:12:29.495456 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:29.495405    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz" podStartSLOduration=5.495392002 podStartE2EDuration="5.495392002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:12:29.494262362 +0000 UTC m=+1368.241451893" watchObservedRunningTime="2026-10-31 23:12:29.495392002 +0000 UTC m=+1368.242581506"
Oct 31 23:12:29.517178 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:29.517126    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq" podStartSLOduration=5.517112884 podStartE2EDuration="5.517112884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:12:29.517035861 +0000 UTC m=+1368.264225367" watchObservedRunningTime="2026-10-31 23:12:29.517112884 +0000 UTC m=+1368.264302390"
Oct 31 23:12:29.724560 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:29.724483    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:12:29.724560 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:29.724537    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:12:29.725719 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:29.725696    4301 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 23:12:29.725767 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:29.725723    4301 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 23:12:29.725767 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:29.725739    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq" podUID="cc182c32-7804-4e3f-b6eb-d22418055d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 23:12:29.725839 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:29.725758    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz" podUID="3b0d47aa-36ab-46fa-b41f-20a71f2256cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 23:12:29.957383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:29.956914    4301 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 23:12:29.957383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:29.956993    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln" podUID="94a46479-0793-4c0a-8d40-b342c343899c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 23:12:30.724703 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:30.724549    4301 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 23:12:30.725173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:30.725143    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq" podUID="cc182c32-7804-4e3f-b6eb-d22418055d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 23:12:30.725276 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:30.724734    4301 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 23:12:30.725276 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:30.725209    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz" podUID="3b0d47aa-36ab-46fa-b41f-20a71f2256cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 23:12:30.955809 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:30.955774    4301 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 23:12:30.956104 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:30.956076    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln" podUID="94a46479-0793-4c0a-8d40-b342c343899c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 23:12:31.755627 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:31.755578    4301 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 23:12:31.756240 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:31.756191    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq" podUID="cc182c32-7804-4e3f-b6eb-d22418055d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 23:12:31.756917 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:31.756882    4301 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 23:12:31.757094 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:31.757071    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz" podUID="3b0d47aa-36ab-46fa-b41f-20a71f2256cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 23:12:31.956818 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:31.956782    4301 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 23:12:31.959091 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:31.959053    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln" podUID="94a46479-0793-4c0a-8d40-b342c343899c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 23:12:32.619397 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:12:32.618779    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:12:32.724147 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:32.724110    4301 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 23:12:32.724345 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:32.724156    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz" podUID="3b0d47aa-36ab-46fa-b41f-20a71f2256cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 23:12:32.724475 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:32.724408    4301 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 23:12:32.724475 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:32.724456    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq" podUID="cc182c32-7804-4e3f-b6eb-d22418055d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 23:12:32.955850 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:32.955757    4301 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 23:12:32.956450 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:32.956418    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln" podUID="94a46479-0793-4c0a-8d40-b342c343899c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 23:12:33.724887 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:33.724855    4301 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 23:12:33.725646 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:33.725615    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq" podUID="cc182c32-7804-4e3f-b6eb-d22418055d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 23:12:33.725820 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:33.724871    4301 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 23:12:33.725935 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:33.725918    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz" podUID="3b0d47aa-36ab-46fa-b41f-20a71f2256cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 23:12:33.968387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:33.968238    4301 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 23:12:33.968387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:33.968299    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln" podUID="94a46479-0793-4c0a-8d40-b342c343899c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 23:12:34.723929 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:34.723890    4301 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 23:12:34.724115 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:34.723972    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz" podUID="3b0d47aa-36ab-46fa-b41f-20a71f2256cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 23:12:34.724259 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:34.724238    4301 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 23:12:34.724299 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:34.724278    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq" podUID="cc182c32-7804-4e3f-b6eb-d22418055d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 23:12:34.964993 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:34.964960    4301 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 23:12:34.965272 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:34.965245    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln" podUID="94a46479-0793-4c0a-8d40-b342c343899c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 23:12:35.551737 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:35.551671    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"]
Oct 31 23:12:35.557082 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:35.557059    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:12:35.557692 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:35.557434    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"]
Oct 31 23:12:35.562445 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:35.562396    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-135248\"/\"distinct-2-w4-135248-kserve-self-signed-certs\""
Oct 31 23:12:35.637533 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:35.636915    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/595ded14-5c69-427e-a232-e969122cf00f-model-cache\") pod \"distinct-2-w4-135248-kserve-74fc5794d9-wjdzg\" (UID: \"595ded14-5c69-427e-a232-e969122cf00f\") " pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:12:35.637533 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:35.636964    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/595ded14-5c69-427e-a232-e969122cf00f-tmp-dir\") pod \"distinct-2-w4-135248-kserve-74fc5794d9-wjdzg\" (UID: \"595ded14-5c69-427e-a232-e969122cf00f\") " pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:12:35.637533 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:35.637061    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/595ded14-5c69-427e-a232-e969122cf00f-home\") pod \"distinct-2-w4-135248-kserve-74fc5794d9-wjdzg\" (UID: \"595ded14-5c69-427e-a232-e969122cf00f\") " pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:12:35.637533 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:35.637092    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/595ded14-5c69-427e-a232-e969122cf00f-tls-certs\") pod \"distinct-2-w4-135248-kserve-74fc5794d9-wjdzg\" (UID: \"595ded14-5c69-427e-a232-e969122cf00f\") " pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:12:35.637533 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:35.637233    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/595ded14-5c69-427e-a232-e969122cf00f-dshm\") pod \"distinct-2-w4-135248-kserve-74fc5794d9-wjdzg\" (UID: \"595ded14-5c69-427e-a232-e969122cf00f\") " pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:12:35.637533 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:35.637342    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l74mg\" (UniqueName: \"kubernetes.io/projected/595ded14-5c69-427e-a232-e969122cf00f-kube-api-access-l74mg\") pod \"distinct-2-w4-135248-kserve-74fc5794d9-wjdzg\" (UID: \"595ded14-5c69-427e-a232-e969122cf00f\") " pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:12:35.731660 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:35.731629    4301 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 23:12:35.732306 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:35.731911    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz" podUID="3b0d47aa-36ab-46fa-b41f-20a71f2256cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 23:12:35.732850 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:35.732812    4301 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 23:12:35.733016 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:35.732992    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq" podUID="cc182c32-7804-4e3f-b6eb-d22418055d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 23:12:35.762118 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:35.762086    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/595ded14-5c69-427e-a232-e969122cf00f-model-cache\") pod \"distinct-2-w4-135248-kserve-74fc5794d9-wjdzg\" (UID: \"595ded14-5c69-427e-a232-e969122cf00f\") " pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:12:35.762424 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:35.762400    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/595ded14-5c69-427e-a232-e969122cf00f-tmp-dir\") pod \"distinct-2-w4-135248-kserve-74fc5794d9-wjdzg\" (UID: \"595ded14-5c69-427e-a232-e969122cf00f\") " pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:12:35.789391 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:35.782284    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/595ded14-5c69-427e-a232-e969122cf00f-home\") pod \"distinct-2-w4-135248-kserve-74fc5794d9-wjdzg\" (UID: \"595ded14-5c69-427e-a232-e969122cf00f\") " pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:12:35.789726 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:35.789701    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/595ded14-5c69-427e-a232-e969122cf00f-tls-certs\") pod \"distinct-2-w4-135248-kserve-74fc5794d9-wjdzg\" (UID: \"595ded14-5c69-427e-a232-e969122cf00f\") " pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:12:35.790101 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:35.790068    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/595ded14-5c69-427e-a232-e969122cf00f-dshm\") pod \"distinct-2-w4-135248-kserve-74fc5794d9-wjdzg\" (UID: \"595ded14-5c69-427e-a232-e969122cf00f\") " pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:12:35.793439 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:35.782895    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/595ded14-5c69-427e-a232-e969122cf00f-home\") pod \"distinct-2-w4-135248-kserve-74fc5794d9-wjdzg\" (UID: \"595ded14-5c69-427e-a232-e969122cf00f\") " pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:12:35.800579 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:35.800541    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l74mg\" (UniqueName: \"kubernetes.io/projected/595ded14-5c69-427e-a232-e969122cf00f-kube-api-access-l74mg\") pod \"distinct-2-w4-135248-kserve-74fc5794d9-wjdzg\" (UID: \"595ded14-5c69-427e-a232-e969122cf00f\") " pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:12:35.806447 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:35.782086    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/595ded14-5c69-427e-a232-e969122cf00f-tmp-dir\") pod \"distinct-2-w4-135248-kserve-74fc5794d9-wjdzg\" (UID: \"595ded14-5c69-427e-a232-e969122cf00f\") " pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:12:35.813420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:35.770319    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/595ded14-5c69-427e-a232-e969122cf00f-model-cache\") pod \"distinct-2-w4-135248-kserve-74fc5794d9-wjdzg\" (UID: \"595ded14-5c69-427e-a232-e969122cf00f\") " pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:12:35.900413 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:35.896881    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/595ded14-5c69-427e-a232-e969122cf00f-dshm\") pod \"distinct-2-w4-135248-kserve-74fc5794d9-wjdzg\" (UID: \"595ded14-5c69-427e-a232-e969122cf00f\") " pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:12:35.970251 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:35.970186    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l74mg\" (UniqueName: \"kubernetes.io/projected/595ded14-5c69-427e-a232-e969122cf00f-kube-api-access-l74mg\") pod \"distinct-2-w4-135248-kserve-74fc5794d9-wjdzg\" (UID: \"595ded14-5c69-427e-a232-e969122cf00f\") " pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:12:35.985745 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:35.985087    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/595ded14-5c69-427e-a232-e969122cf00f-tls-certs\") pod \"distinct-2-w4-135248-kserve-74fc5794d9-wjdzg\" (UID: \"595ded14-5c69-427e-a232-e969122cf00f\") " pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:12:36.248488 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:36.248338    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:12:36.297264 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:36.294939    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:12:36.297264 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:36.295386    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:12:36.298248 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:36.298225    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:12:36.617458 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:36.617423    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"]
Oct 31 23:12:36.618417 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:12:36.618352    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod595ded14_5c69_427e_a232_e969122cf00f.slice/crio-573b9ce04339a4a89e16c1ed87aa27918b0235bcf2f6b4758348056aa5d1e48b WatchSource:0}: Error finding container 573b9ce04339a4a89e16c1ed87aa27918b0235bcf2f6b4758348056aa5d1e48b: Status 404 returned error can't find the container with id 573b9ce04339a4a89e16c1ed87aa27918b0235bcf2f6b4758348056aa5d1e48b
Oct 31 23:12:36.725429 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:36.725104    4301 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 23:12:36.725429 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:36.725155    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq" podUID="cc182c32-7804-4e3f-b6eb-d22418055d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 23:12:36.726481 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:36.726402    4301 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 23:12:36.726481 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:36.726439    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz" podUID="3b0d47aa-36ab-46fa-b41f-20a71f2256cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 23:12:36.888918 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:36.888880    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" event={"ID":"595ded14-5c69-427e-a232-e969122cf00f","Type":"ContainerStarted","Data":"03db228446522bf212a08d53b5d2d1d60542e2a3767b08327929de6e374b0f3d"}
Oct 31 23:12:36.888918 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:36.888923    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" event={"ID":"595ded14-5c69-427e-a232-e969122cf00f","Type":"ContainerStarted","Data":"573b9ce04339a4a89e16c1ed87aa27918b0235bcf2f6b4758348056aa5d1e48b"}
Oct 31 23:12:37.248878 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:37.248782    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:12:37.728227 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:37.728198    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:12:37.728721 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:37.728268    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:12:37.728937 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:37.728915    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:12:37.729023 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:37.728962    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:12:37.729597 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:37.729578    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:12:37.729700 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:37.729640    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:12:37.757781 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:37.757735    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podStartSLOduration=2.757721321 podStartE2EDuration="2.757721321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:12:36.919837178 +0000 UTC m=+1375.667026685" watchObservedRunningTime="2026-10-31 23:12:37.757721321 +0000 UTC m=+1376.504910889"
Oct 31 23:12:39.707618 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.707589    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"]
Oct 31 23:12:39.711650 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.711632    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:12:39.726857 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.726830    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"]
Oct 31 23:12:39.767407 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.766526    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"]
Oct 31 23:12:39.772206 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.772178    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"]
Oct 31 23:12:39.772384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.772313    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:12:39.793310 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.793254    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkm4t\" (UniqueName: \"kubernetes.io/projected/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-kube-api-access-gkm4t\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:12:39.793481 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.793395    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istio-token\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:12:39.793544 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.793482    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-workload-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:12:39.793544 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.793526    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-istio-data\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:12:39.793638 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.793553    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a96c54a-c66d-466f-b2d4-84a208c3c664-istio-podinfo\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:12:39.793758 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.793736    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a96c54a-c66d-466f-b2d4-84a208c3c664-istio-token\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:12:39.793860 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.793788    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-workload-certs\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:12:39.793860 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.793830    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-credential-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:12:39.793860 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.793854    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjj5s\" (UniqueName: \"kubernetes.io/projected/7a96c54a-c66d-466f-b2d4-84a208c3c664-kube-api-access-zjj5s\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:12:39.794005 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.793874    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a96c54a-c66d-466f-b2d4-84a208c3c664-istiod-ca-cert\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:12:39.794005 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.793913    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istio-data\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:12:39.794005 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.793930    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-workload-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:12:39.794005 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.793944    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istiod-ca-cert\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:12:39.794005 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.793971    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-credential-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:12:39.794208 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.794049    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istio-envoy\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:12:39.794208 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.794074    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istio-podinfo\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:12:39.794208 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.794095    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-workload-certs\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:12:39.794208 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.794169    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-istio-envoy\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:12:39.895722 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.895695    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a96c54a-c66d-466f-b2d4-84a208c3c664-istiod-ca-cert\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:12:39.895899 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.895730    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istio-data\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:12:39.895899 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.895751    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-workload-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:12:39.895899 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.895768    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istiod-ca-cert\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:12:39.895899 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.895800    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-credential-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:12:39.895899 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.895897    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istio-envoy\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:12:39.896171 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.895924    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istio-podinfo\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:12:39.896171 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.895945    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-workload-certs\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:12:39.896171 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.896014    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-istio-envoy\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:12:39.896171 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.896050    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkm4t\" (UniqueName: \"kubernetes.io/projected/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-kube-api-access-gkm4t\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:12:39.896171 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.896157    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istio-token\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:12:39.896474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.896181    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-workload-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:12:39.896474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.896182    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istio-data\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:12:39.896474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.896212    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-istio-data\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:12:39.896474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.896237    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a96c54a-c66d-466f-b2d4-84a208c3c664-istio-podinfo\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:12:39.896474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.896258    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-workload-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:12:39.896474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.896287    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a96c54a-c66d-466f-b2d4-84a208c3c664-istio-token\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:12:39.896474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.896328    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-workload-certs\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:12:39.896474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.896397    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-credential-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:12:39.896474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.896442    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjj5s\" (UniqueName: \"kubernetes.io/projected/7a96c54a-c66d-466f-b2d4-84a208c3c664-kube-api-access-zjj5s\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:12:39.897498 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.896540    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a96c54a-c66d-466f-b2d4-84a208c3c664-istiod-ca-cert\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:12:39.897498 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.896584    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-workload-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:12:39.897498 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.896704    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istiod-ca-cert\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:12:39.897498 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.896806    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-credential-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:12:39.897498 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.896816    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-workload-certs\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:12:39.897498 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.896983    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-workload-certs\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:12:39.897498 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.897056    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-credential-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:12:39.897498 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.897251    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-istio-data\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:12:39.898711 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.898687    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istio-envoy\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:12:39.899092 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.899073    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a96c54a-c66d-466f-b2d4-84a208c3c664-istio-podinfo\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:12:39.899424 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.899401    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-istio-envoy\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:12:39.899854 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.899833    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istio-podinfo\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:12:39.910610 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.910578    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjj5s\" (UniqueName: \"kubernetes.io/projected/7a96c54a-c66d-466f-b2d4-84a208c3c664-kube-api-access-zjj5s\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:12:39.910737 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.910656    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a96c54a-c66d-466f-b2d4-84a208c3c664-istio-token\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:12:39.910855 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.910831    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istio-token\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:12:39.911002 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:39.910982    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkm4t\" (UniqueName: \"kubernetes.io/projected/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-kube-api-access-gkm4t\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:12:40.027969 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:40.027890    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:12:40.090633 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:40.090610    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:12:40.323414 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:40.323388    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"]
Oct 31 23:12:40.324757 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:12:40.324730    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a96c54a_c66d_466f_b2d4_84a208c3c664.slice/crio-8d7d8e2468f5286ba022818ca4c1b081230f64db297bdc28be77ca75954f9b75 WatchSource:0}: Error finding container 8d7d8e2468f5286ba022818ca4c1b081230f64db297bdc28be77ca75954f9b75: Status 404 returned error can't find the container with id 8d7d8e2468f5286ba022818ca4c1b081230f64db297bdc28be77ca75954f9b75
Oct 31 23:12:40.327326 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:40.327297    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:12:40.327429 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:40.327375    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:12:40.327429 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:40.327411    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:12:40.407140 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:40.407108    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"]
Oct 31 23:12:40.407393 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:12:40.407120    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c87d0a_e7ea_44ed_a02d_c92554d9f41e.slice/crio-40df41ab770e7201fbc719db54c127663a3f2ef6b8c5a3a09c1a7ce9c52e0c06 WatchSource:0}: Error finding container 40df41ab770e7201fbc719db54c127663a3f2ef6b8c5a3a09c1a7ce9c52e0c06: Status 404 returned error can't find the container with id 40df41ab770e7201fbc719db54c127663a3f2ef6b8c5a3a09c1a7ce9c52e0c06
Oct 31 23:12:40.410078 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:40.410046    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:12:40.410192 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:40.410120    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:12:40.410192 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:40.410175    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:12:40.919693 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:40.919648    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5" event={"ID":"7a96c54a-c66d-466f-b2d4-84a208c3c664","Type":"ContainerStarted","Data":"44749bf5ac0653dd24caea25c2aae582b8055a50e26be94d07449f9f0ae54358"}
Oct 31 23:12:40.920298 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:40.919731    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5" event={"ID":"7a96c54a-c66d-466f-b2d4-84a208c3c664","Type":"ContainerStarted","Data":"8d7d8e2468f5286ba022818ca4c1b081230f64db297bdc28be77ca75954f9b75"}
Oct 31 23:12:40.921502 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:40.921469    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl" event={"ID":"92c87d0a-e7ea-44ed-a02d-c92554d9f41e","Type":"ContainerStarted","Data":"e39e6911cc519a121d4efa4339de5d4517d67781b9156b9d2ba4ad370a03dfa2"}
Oct 31 23:12:40.921638 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:40.921519    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl" event={"ID":"92c87d0a-e7ea-44ed-a02d-c92554d9f41e","Type":"ContainerStarted","Data":"40df41ab770e7201fbc719db54c127663a3f2ef6b8c5a3a09c1a7ce9c52e0c06"}
Oct 31 23:12:40.986220 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:40.984226    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl" podStartSLOduration=1.98420442 podStartE2EDuration="1.98420442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:12:40.983269552 +0000 UTC m=+1379.730459057" watchObservedRunningTime="2026-10-31 23:12:40.98420442 +0000 UTC m=+1379.731393927"
Oct 31 23:12:40.993969 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:40.993899    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5" podStartSLOduration=1.9938724460000001 podStartE2EDuration="1.993872446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:12:40.944699845 +0000 UTC m=+1379.691889354" watchObservedRunningTime="2026-10-31 23:12:40.993872446 +0000 UTC m=+1379.741061952"
Oct 31 23:12:41.050380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:41.049512    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:12:41.050380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:41.049705    4301 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 31 23:12:41.050380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:41.049752    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5" podUID="7a96c54a-c66d-466f-b2d4-84a208c3c664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 31 23:12:41.092265 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:41.092186    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:12:41.093930 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:41.093907    4301 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 23:12:41.094030 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:41.093956    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl" podUID="92c87d0a-e7ea-44ed-a02d-c92554d9f41e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 23:12:42.034384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:42.032745    4301 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 31 23:12:42.034384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:42.032804    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5" podUID="7a96c54a-c66d-466f-b2d4-84a208c3c664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 31 23:12:42.092129 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:42.092097    4301 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 23:12:42.092293 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:42.092149    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl" podUID="92c87d0a-e7ea-44ed-a02d-c92554d9f41e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 23:12:43.036527 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:43.036490    4301 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 31 23:12:43.037144 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:43.037113    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5" podUID="7a96c54a-c66d-466f-b2d4-84a208c3c664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 31 23:12:43.092773 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:43.092741    4301 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 23:12:43.092951 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:43.092795    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl" podUID="92c87d0a-e7ea-44ed-a02d-c92554d9f41e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 23:12:44.028729 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:44.028689    4301 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 31 23:12:44.028982 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:44.028950    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5" podUID="7a96c54a-c66d-466f-b2d4-84a208c3c664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 31 23:12:44.092140 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:44.092103    4301 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 23:12:44.092803 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:44.092771    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl" podUID="92c87d0a-e7ea-44ed-a02d-c92554d9f41e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 23:12:45.038394 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:45.030855    4301 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 31 23:12:45.038394 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:45.030915    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5" podUID="7a96c54a-c66d-466f-b2d4-84a208c3c664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 31 23:12:45.097384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:45.096749    4301 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 23:12:45.097384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:45.096809    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl" podUID="92c87d0a-e7ea-44ed-a02d-c92554d9f41e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 23:12:46.028243 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:46.028213    4301 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 31 23:12:46.028507 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:46.028267    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5" podUID="7a96c54a-c66d-466f-b2d4-84a208c3c664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 31 23:12:46.092741 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:46.092697    4301 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 23:12:46.092947 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:46.092755    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl" podUID="92c87d0a-e7ea-44ed-a02d-c92554d9f41e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 23:12:46.256387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:46.255403    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:12:46.308305 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:46.304237    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:12:47.028315 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:47.028277    4301 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 31 23:12:47.028592 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:47.028336    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5" podUID="7a96c54a-c66d-466f-b2d4-84a208c3c664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 31 23:12:47.040033 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:47.040009    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:12:47.106379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:47.105752    4301 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 23:12:47.106379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:47.105817    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl" podUID="92c87d0a-e7ea-44ed-a02d-c92554d9f41e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 23:12:48.453133 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:48.453099    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:12:48.454820 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:48.454676    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:12:48.460715 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:48.460684    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:12:48.569378 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:48.567713    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:12:48.569547 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:48.569434    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:12:48.585084 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:48.582695    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:12:54.260463 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:54.259977    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:54.342386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:54.341474    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:54.372535 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:54.372482    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:54.427506 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:54.427455    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:54.578453 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:54.578346    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:55.026511 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:55.026460    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:55.103489 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:55.102479    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:55.272542 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:55.272485    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 23:12:55.314581 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:55.314486    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:55.319388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:55.315351    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:12:55.452388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:55.449025    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:55.452388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:55.382534    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:55.452388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:55.452254    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 23:12:55.452739 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:55.452590    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:55.589452 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:55.584788    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded"
Oct 31 23:12:55.696386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:55.684157    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:55.777386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:55.774908    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:12:55.777386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:55.774971    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:12:55.777386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:55.775004    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:12:56.057204 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:56.057155    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:56.064386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:56.060477    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:12:56.111947 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:56.111890    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:56.130841 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:56.130811    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:12:56.131259 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:56.131245    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:12:56.343435 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:56.338597    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 23:12:56.373691 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:56.373648    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 23:12:56.582384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:56.540381    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:12:56.587379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:56.583413    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 31 23:12:56.587379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:56.584181    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 31 23:12:56.587379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:56.585009    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 23:12:56.587379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:56.587065    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:12:56.647933 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:56.647898    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 31 23:12:56.698470 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:56.698414    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:56.699997 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:56.699974    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:12:57.014597 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:57.014488    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded"
Oct 31 23:12:57.072466 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:57.069928    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:57.155701 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:57.155653    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:57.156378 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:57.156327    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:57.341574 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:57.341462    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:57.406806 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:57.374209    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Oct 31 23:12:57.406806 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:57.374446    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:12:57.406806 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:57.383613    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:57.406806 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:57.390404    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:57.601864 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:57.577138    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:12:57.601864 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:57.599625    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:57.601864 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:57.599962    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 31 23:12:57.680453 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:57.659810    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:57.680453 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:57.660223    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:12:58.073496 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:58.073449    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:58.118585 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:58.118537    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:58.163415 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:58.158727    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 23:12:58.163415 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:58.159906    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 23:12:58.163415 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:58.160798    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:12:58.361702 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:58.361494    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:58.389209 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:58.388918    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:58.474097 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:58.474068    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:12:58.622493 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:58.615318    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:58.642819 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:58.642773    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:12:58.644124 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:58.622859    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:58.673500 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:58.673441    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:12:58.733377 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:58.719271    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:12:58.846379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:58.845946    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:12:58.876545 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:58.876462    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:12:58.895120 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:58.895095    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:12:58.901858 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:58.901837    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:12:58.904509 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:12:58.904481    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:13:02.548033 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:13:02.547990    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:13:09.477743 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.477708    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"]
Oct 31 23:13:09.482695 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.482671    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:13:09.500534 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.497663    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"]
Oct 31 23:13:09.535892 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.535864    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85e9dd14-aa39-400e-b672-4c3122ea568a-istiod-ca-cert\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:13:09.536069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.535905    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85e9dd14-aa39-400e-b672-4c3122ea568a-istio-podinfo\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:13:09.536144 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.536109    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-workload-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:13:09.536251 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.536219    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-credential-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:13:09.536313 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.536266    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j5w8\" (UniqueName: \"kubernetes.io/projected/85e9dd14-aa39-400e-b672-4c3122ea568a-kube-api-access-7j5w8\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:13:09.536369 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.536319    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85e9dd14-aa39-400e-b672-4c3122ea568a-istio-token\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:13:09.536436 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.536412    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-workload-certs\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:13:09.536477 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.536447    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-istio-envoy\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:13:09.536559 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.536543    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-istio-data\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:13:09.613353 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.613321    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"]
Oct 31 23:13:09.617338 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.617317    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:13:09.637862 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.637832    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-workload-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:13:09.638021 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.637894    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-credential-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:13:09.638021 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.637916    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j5w8\" (UniqueName: \"kubernetes.io/projected/85e9dd14-aa39-400e-b672-4c3122ea568a-kube-api-access-7j5w8\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:13:09.638021 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.637955    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85e9dd14-aa39-400e-b672-4c3122ea568a-istio-token\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:13:09.638021 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.637995    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-workload-certs\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:13:09.638276 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.638027    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-istio-envoy\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:13:09.638276 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.638117    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-istio-data\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:13:09.638276 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.638188    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85e9dd14-aa39-400e-b672-4c3122ea568a-istiod-ca-cert\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:13:09.638276 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.638222    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85e9dd14-aa39-400e-b672-4c3122ea568a-istio-podinfo\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:13:09.638276 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.638263    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-workload-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:13:09.639603 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.638496    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-credential-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:13:09.639603 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.638796    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-istio-data\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:13:09.639603 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.638960    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-workload-certs\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:13:09.640340 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.640313    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85e9dd14-aa39-400e-b672-4c3122ea568a-istiod-ca-cert\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:13:09.641114 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.641091    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-istio-envoy\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:13:09.641484 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.641463    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85e9dd14-aa39-400e-b672-4c3122ea568a-istio-podinfo\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:13:09.641565 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.641501    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"]
Oct 31 23:13:09.661166 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.661113    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j5w8\" (UniqueName: \"kubernetes.io/projected/85e9dd14-aa39-400e-b672-4c3122ea568a-kube-api-access-7j5w8\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:13:09.661572 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.661547    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85e9dd14-aa39-400e-b672-4c3122ea568a-istio-token\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:13:09.739228 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.739151    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-workload-certs\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:13:09.739228 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.739183    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-istio-data\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:13:09.739532 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.739283    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-workload-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:13:09.739532 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.739335    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/944451b3-0e62-4419-bf3a-37401268d08f-istiod-ca-cert\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:13:09.739532 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.739470    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/944451b3-0e62-4419-bf3a-37401268d08f-istio-podinfo\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:13:09.739655 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.739612    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-credential-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:13:09.739690 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.739663    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-istio-envoy\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:13:09.739735 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.739690    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/944451b3-0e62-4419-bf3a-37401268d08f-istio-token\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:13:09.739799 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.739785    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dfl9\" (UniqueName: \"kubernetes.io/projected/944451b3-0e62-4419-bf3a-37401268d08f-kube-api-access-2dfl9\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:13:09.803158 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.803128    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:13:09.841563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.841522    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-workload-certs\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:13:09.841563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.841561    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-istio-data\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:13:09.841822 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.841635    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-workload-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:13:09.841822 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.841667    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/944451b3-0e62-4419-bf3a-37401268d08f-istiod-ca-cert\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:13:09.841822 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.841806    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/944451b3-0e62-4419-bf3a-37401268d08f-istio-podinfo\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:13:09.841965 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.841947    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-credential-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:13:09.842001 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.841981    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-istio-envoy\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:13:09.842001 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.841997    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/944451b3-0e62-4419-bf3a-37401268d08f-istio-token\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:13:09.842069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.842030    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-workload-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:13:09.842069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.842053    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dfl9\" (UniqueName: \"kubernetes.io/projected/944451b3-0e62-4419-bf3a-37401268d08f-kube-api-access-2dfl9\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:13:09.842147 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.842131    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-istio-data\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:13:09.842302 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.842282    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-credential-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:13:09.842405 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.842052    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-workload-certs\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:13:09.842491 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.842461    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/944451b3-0e62-4419-bf3a-37401268d08f-istiod-ca-cert\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:13:09.844317 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.844293    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-istio-envoy\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:13:09.844715 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.844694    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/944451b3-0e62-4419-bf3a-37401268d08f-istio-podinfo\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:13:09.861228 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.861201    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dfl9\" (UniqueName: \"kubernetes.io/projected/944451b3-0e62-4419-bf3a-37401268d08f-kube-api-access-2dfl9\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:13:09.865659 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.865166    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/944451b3-0e62-4419-bf3a-37401268d08f-istio-token\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:13:09.934536 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:09.934504    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:13:10.121256 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:10.117095    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"]
Oct 31 23:13:10.125335 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:13:10.125308    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85e9dd14_aa39_400e_b672_4c3122ea568a.slice/crio-7c5bf25d497937f98f53c562d7ff56f91928b68a630de341ff323896e3c04505 WatchSource:0}: Error finding container 7c5bf25d497937f98f53c562d7ff56f91928b68a630de341ff323896e3c04505: Status 404 returned error can't find the container with id 7c5bf25d497937f98f53c562d7ff56f91928b68a630de341ff323896e3c04505
Oct 31 23:13:10.127697 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:10.127659    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:13:10.127795 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:10.127750    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:13:10.127851 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:10.127805    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:13:10.267331 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:10.267304    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"]
Oct 31 23:13:10.270155 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:13:10.270071    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944451b3_0e62_4419_bf3a_37401268d08f.slice/crio-9fa108f7d512f5d0f0a126a321daa4903a0c48314c76c623104f9ca066e9cd3d WatchSource:0}: Error finding container 9fa108f7d512f5d0f0a126a321daa4903a0c48314c76c623104f9ca066e9cd3d: Status 404 returned error can't find the container with id 9fa108f7d512f5d0f0a126a321daa4903a0c48314c76c623104f9ca066e9cd3d
Oct 31 23:13:10.272726 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:10.272691    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:13:10.272843 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:10.272758    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:13:10.272843 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:10.272790    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:13:10.939086 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:10.939037    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs" event={"ID":"944451b3-0e62-4419-bf3a-37401268d08f","Type":"ContainerStarted","Data":"fb09bceafe3fe9ee13ce54940d702896cd7be3b03819ead1f800208d4b19f688"}
Oct 31 23:13:10.939630 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:10.939107    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs" event={"ID":"944451b3-0e62-4419-bf3a-37401268d08f","Type":"ContainerStarted","Data":"9fa108f7d512f5d0f0a126a321daa4903a0c48314c76c623104f9ca066e9cd3d"}
Oct 31 23:13:10.942189 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:10.942156    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g" event={"ID":"85e9dd14-aa39-400e-b672-4c3122ea568a","Type":"ContainerStarted","Data":"febad0a1c9c7e1bd6fe226729255932abb68d1788e8ece571e9a507aceac5979"}
Oct 31 23:13:10.942431 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:10.942412    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g" event={"ID":"85e9dd14-aa39-400e-b672-4c3122ea568a","Type":"ContainerStarted","Data":"7c5bf25d497937f98f53c562d7ff56f91928b68a630de341ff323896e3c04505"}
Oct 31 23:13:11.010476 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:11.010323    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs" podStartSLOduration=2.010303758 podStartE2EDuration="2.010303758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:13:10.977853668 +0000 UTC m=+1409.725043188" watchObservedRunningTime="2026-10-31 23:13:11.010303758 +0000 UTC m=+1409.757493274"
Oct 31 23:13:11.013109 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:11.013062    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g" podStartSLOduration=2.013044038 podStartE2EDuration="2.013044038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:13:11.007528334 +0000 UTC m=+1409.754717862" watchObservedRunningTime="2026-10-31 23:13:11.013044038 +0000 UTC m=+1409.760233545"
Oct 31 23:13:11.803758 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:11.803723    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:13:11.805524 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:11.805498    4301 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 31 23:13:11.805617 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:11.805551    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g" podUID="85e9dd14-aa39-400e-b672-4c3122ea568a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 31 23:13:11.945385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:11.943006    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:13:11.958382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:11.947219    4301 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 23:13:11.958382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:11.955454    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs" podUID="944451b3-0e62-4419-bf3a-37401268d08f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 23:13:12.803919 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:12.803885    4301 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 31 23:13:12.804110 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:12.803935    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g" podUID="85e9dd14-aa39-400e-b672-4c3122ea568a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 31 23:13:12.947380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:12.937779    4301 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 23:13:12.947380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:12.937839    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs" podUID="944451b3-0e62-4419-bf3a-37401268d08f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 23:13:13.804060 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:13.804012    4301 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 31 23:13:13.804249 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:13.804093    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g" podUID="85e9dd14-aa39-400e-b672-4c3122ea568a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 31 23:13:13.944770 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:13.944735    4301 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 23:13:13.945020 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:13.944992    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs" podUID="944451b3-0e62-4419-bf3a-37401268d08f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 23:13:14.803573 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:14.803542    4301 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 31 23:13:14.803992 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:14.803593    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g" podUID="85e9dd14-aa39-400e-b672-4c3122ea568a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 31 23:13:14.948865 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:14.948825    4301 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 23:13:14.949198 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:14.949156    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs" podUID="944451b3-0e62-4419-bf3a-37401268d08f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 23:13:15.514965 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:15.514933    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:13:15.804021 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:15.803942    4301 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 31 23:13:15.804021 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:15.803992    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g" podUID="85e9dd14-aa39-400e-b672-4c3122ea568a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 31 23:13:15.936763 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:15.936731    4301 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 23:13:15.937032 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:15.937000    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs" podUID="944451b3-0e62-4419-bf3a-37401268d08f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 23:13:16.807210 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:16.806766    4301 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 31 23:13:16.807210 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:16.806826    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g" podUID="85e9dd14-aa39-400e-b672-4c3122ea568a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 31 23:13:16.935771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:16.935737    4301 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 23:13:16.936066 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:16.936036    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs" podUID="944451b3-0e62-4419-bf3a-37401268d08f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 23:13:17.956384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:17.825217    4301 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 31 23:13:17.956384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:17.825304    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g" podUID="85e9dd14-aa39-400e-b672-4c3122ea568a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 31 23:13:18.010941 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:17.985029    4301 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 23:13:18.050611 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:18.050554    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs" podUID="944451b3-0e62-4419-bf3a-37401268d08f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 23:13:18.733384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:18.729534    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:18.768382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:18.761964    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 31 23:13:18.801393 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:18.800477    4301 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 23:13:18.801393 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:18.800549    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954" podUID="929c3031-62b4-4587-9f53-75de251130c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:19.086635 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:19.086540    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:19.382565 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:19.382524    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:19.393897 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:19.393844    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:19.774706 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:19.774606    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 31 23:13:19.825426 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:19.807079    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:13:19.835736 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:19.834954    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 31 23:13:19.836206 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:19.835113    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 31 23:13:19.836966 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:19.836946    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:13:19.876884 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:19.876839    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:19.877518 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:19.877484    4301 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 23:13:19.878726 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:19.878693    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g" podUID="85e9dd14-aa39-400e-b672-4c3122ea568a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:20.043676 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:20.043571    4301 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 23:13:20.043957 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:20.043927    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs" podUID="944451b3-0e62-4419-bf3a-37401268d08f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:20.057831 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:20.057797    4301 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 23:13:20.058104 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:20.058075    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t" podUID="fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:20.078534 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:20.078487    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:20.115432 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:20.115389    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:20.240754 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:20.212261    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 23:13:20.241327 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:20.241301    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:13:20.359707 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:20.358471    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:20.533591 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:20.533529    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:20.577385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:20.565484    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:13:20.577385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:20.566503    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:20.577385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:20.567435    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:13:20.577385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:20.569209    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:20.846389 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:20.808501    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:20.847959 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:20.846742    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:20.853394 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:20.849295    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 31 23:13:20.861390 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:20.854227    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:13:20.920408 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:20.884661    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 23:13:20.920408 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:20.884861    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:13:20.925930 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:20.925892    4301 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 23:13:20.926187 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:20.926154    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g" podUID="85e9dd14-aa39-400e-b672-4c3122ea568a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:21.084424 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:21.075402    4301 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 23:13:21.084872 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:21.084714    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs" podUID="944451b3-0e62-4419-bf3a-37401268d08f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:21.093284 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:21.093237    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:21.101985 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:21.094997    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": context deadline exceeded"
Oct 31 23:13:21.101985 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:21.095116    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:13:21.270343 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:21.270290    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:21.324847 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:21.281432    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 23:13:21.325787 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:21.281575    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:13:21.325787 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:21.325627    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:13:21.325787 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:21.325681    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:13:21.333793 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:21.333719    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:13:21.369122 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:21.367624    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:21.373380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:21.369465    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:13:21.373380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:21.371122    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:13:21.607333 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:21.607286    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:21.707334 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:21.707234    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:13:21.707661 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:21.707643    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:13:21.707824 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:21.707811    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:13:21.856508 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:21.856455    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:21.934106 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:21.934067    4301 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 23:13:21.956406 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:21.947425    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g" podUID="85e9dd14-aa39-400e-b672-4c3122ea568a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:21.973329 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:21.970557    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:22.041470 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:22.022106    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:13:22.073394 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:22.071493    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:13:22.135381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:22.129663    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 23:13:22.193612 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:22.193582    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:13:22.199925 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:22.199898    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:13:22.231718 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:22.231636    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:13:22.234782 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:22.233461    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:13:22.996539 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:22.996498    4301 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 23:13:22.997193 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:22.997158    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g" podUID="85e9dd14-aa39-400e-b672-4c3122ea568a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:23.342169 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:23.342095    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:13:23.343391 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:23.343372    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:13:23.344086 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:23.344066    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:13:31.197887 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:31.197852    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8e05f76-jw5n4"]
Oct 31 23:13:31.216740 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:31.216709    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e05f76-jw5n4"
Oct 31 23:13:31.223735 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:31.223707    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\""
Oct 31 23:13:31.223964 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:31.223949    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 23:13:31.247410 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:31.247142    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8e05f76-jw5n4"]
Oct 31 23:13:31.262760 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:31.262730    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbf2a79d-5e58-490e-9947-68226f151f1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8e05f76-jw5n4\" (UID: \"dbf2a79d-5e58-490e-9947-68226f151f1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e05f76-jw5n4"
Oct 31 23:13:31.262989 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:31.262968    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbf2a79d-5e58-490e-9947-68226f151f1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8e05f76-jw5n4\" (UID: \"dbf2a79d-5e58-490e-9947-68226f151f1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e05f76-jw5n4"
Oct 31 23:13:31.263086 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:31.263070    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55q5b\" (UniqueName: \"kubernetes.io/projected/dbf2a79d-5e58-490e-9947-68226f151f1f-kube-api-access-55q5b\") pod \"maas-api-revoke-sub-c8e05f76-jw5n4\" (UID: \"dbf2a79d-5e58-490e-9947-68226f151f1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e05f76-jw5n4"
Oct 31 23:13:31.364427 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:31.364336    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbf2a79d-5e58-490e-9947-68226f151f1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8e05f76-jw5n4\" (UID: \"dbf2a79d-5e58-490e-9947-68226f151f1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e05f76-jw5n4"
Oct 31 23:13:31.364604 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:31.364506    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbf2a79d-5e58-490e-9947-68226f151f1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8e05f76-jw5n4\" (UID: \"dbf2a79d-5e58-490e-9947-68226f151f1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e05f76-jw5n4"
Oct 31 23:13:31.364604 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:31.364543    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55q5b\" (UniqueName: \"kubernetes.io/projected/dbf2a79d-5e58-490e-9947-68226f151f1f-kube-api-access-55q5b\") pod \"maas-api-revoke-sub-c8e05f76-jw5n4\" (UID: \"dbf2a79d-5e58-490e-9947-68226f151f1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e05f76-jw5n4"
Oct 31 23:13:31.365192 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:31.365171    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbf2a79d-5e58-490e-9947-68226f151f1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8e05f76-jw5n4\" (UID: \"dbf2a79d-5e58-490e-9947-68226f151f1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e05f76-jw5n4"
Oct 31 23:13:31.366749 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:31.366726    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbf2a79d-5e58-490e-9947-68226f151f1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8e05f76-jw5n4\" (UID: \"dbf2a79d-5e58-490e-9947-68226f151f1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e05f76-jw5n4"
Oct 31 23:13:31.375814 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:31.375783    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55q5b\" (UniqueName: \"kubernetes.io/projected/dbf2a79d-5e58-490e-9947-68226f151f1f-kube-api-access-55q5b\") pod \"maas-api-revoke-sub-c8e05f76-jw5n4\" (UID: \"dbf2a79d-5e58-490e-9947-68226f151f1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e05f76-jw5n4"
Oct 31 23:13:31.562074 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:31.561998    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e05f76-jw5n4"
Oct 31 23:13:31.849314 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:31.849289    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8e05f76-jw5n4"]
Oct 31 23:13:31.851702 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:13:31.851664    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf2a79d_5e58_490e_9947_68226f151f1f.slice/crio-d126d519b24f4581dbe5f5b113563958050e72a3b26145f8553faf442af6f520 WatchSource:0}: Error finding container d126d519b24f4581dbe5f5b113563958050e72a3b26145f8553faf442af6f520: Status 404 returned error can't find the container with id d126d519b24f4581dbe5f5b113563958050e72a3b26145f8553faf442af6f520
Oct 31 23:13:32.003404 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:31.996405    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e05f76-jw5n4" event={"ID":"dbf2a79d-5e58-490e-9947-68226f151f1f","Type":"ContainerStarted","Data":"d126d519b24f4581dbe5f5b113563958050e72a3b26145f8553faf442af6f520"}
Oct 31 23:13:32.549656 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:13:32.549558    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:13:33.005016 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:33.004981    4301 generic.go:356] "Generic (PLEG): container finished" podID="dbf2a79d-5e58-490e-9947-68226f151f1f" containerID="215edbd41a4bd8e4963941427e1de3418a0499f65810ce15eefaa1a9a6e7bb42" exitCode=0
Oct 31 23:13:33.005177 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:33.005050    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e05f76-jw5n4" event={"ID":"dbf2a79d-5e58-490e-9947-68226f151f1f","Type":"ContainerDied","Data":"215edbd41a4bd8e4963941427e1de3418a0499f65810ce15eefaa1a9a6e7bb42"}
Oct 31 23:13:34.251838 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:34.251804    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e05f76-jw5n4"
Oct 31 23:13:34.390217 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:34.390184    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbf2a79d-5e58-490e-9947-68226f151f1f-kube-api-access-55q5b\" (UniqueName: \"kubernetes.io/projected/dbf2a79d-5e58-490e-9947-68226f151f1f-kube-api-access-55q5b\") pod \"dbf2a79d-5e58-490e-9947-68226f151f1f\" (UID: \"dbf2a79d-5e58-490e-9947-68226f151f1f\") "
Oct 31 23:13:34.390397 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:34.390381    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbf2a79d-5e58-490e-9947-68226f151f1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbf2a79d-5e58-490e-9947-68226f151f1f-maas-api-openshift-ca\") pod \"dbf2a79d-5e58-490e-9947-68226f151f1f\" (UID: \"dbf2a79d-5e58-490e-9947-68226f151f1f\") "
Oct 31 23:13:34.390491 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:34.390472    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbf2a79d-5e58-490e-9947-68226f151f1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbf2a79d-5e58-490e-9947-68226f151f1f-maas-api-xks-ca\") pod \"dbf2a79d-5e58-490e-9947-68226f151f1f\" (UID: \"dbf2a79d-5e58-490e-9947-68226f151f1f\") "
Oct 31 23:13:34.390778 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:34.390749    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbf2a79d-5e58-490e-9947-68226f151f1f-maas-api-openshift-ca" pod "dbf2a79d-5e58-490e-9947-68226f151f1f" (UID: "dbf2a79d-5e58-490e-9947-68226f151f1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:13:34.390878 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:34.390863    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbf2a79d-5e58-490e-9947-68226f151f1f-maas-api-openshift-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:13:34.392281 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:34.392259    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbf2a79d-5e58-490e-9947-68226f151f1f-kube-api-access-55q5b" pod "dbf2a79d-5e58-490e-9947-68226f151f1f" (UID: "dbf2a79d-5e58-490e-9947-68226f151f1f"). InnerVolumeSpecName "kube-api-access-55q5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:13:34.392474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:34.392456    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbf2a79d-5e58-490e-9947-68226f151f1f-maas-api-xks-ca" pod "dbf2a79d-5e58-490e-9947-68226f151f1f" (UID: "dbf2a79d-5e58-490e-9947-68226f151f1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:13:34.491828 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:34.491788    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55q5b\" (UniqueName: \"kubernetes.io/projected/dbf2a79d-5e58-490e-9947-68226f151f1f-kube-api-access-55q5b\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:13:34.491828 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:34.491825    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbf2a79d-5e58-490e-9947-68226f151f1f-maas-api-xks-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:13:35.017764 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:35.017730    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e05f76-jw5n4" event={"ID":"dbf2a79d-5e58-490e-9947-68226f151f1f","Type":"ContainerDied","Data":"d126d519b24f4581dbe5f5b113563958050e72a3b26145f8553faf442af6f520"}
Oct 31 23:13:35.017943 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:35.017779    4301 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d126d519b24f4581dbe5f5b113563958050e72a3b26145f8553faf442af6f520"
Oct 31 23:13:35.017943 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:35.017770    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e05f76-jw5n4"
Oct 31 23:13:39.659803 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:39.659770    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"]
Oct 31 23:13:39.660373 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:39.660340    4301 cpu_manager.go:414] "containerMap: removing container" podUID="dbf2a79d-5e58-490e-9947-68226f151f1f" containerName="revoke-keys"
Oct 31 23:13:39.660373 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:39.660373    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbf2a79d-5e58-490e-9947-68226f151f1f" containerName="revoke-keys"
Oct 31 23:13:39.660520 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:39.660500    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbf2a79d-5e58-490e-9947-68226f151f1f" containerName="revoke-keys"
Oct 31 23:13:39.663912 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:39.663897    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:13:39.795420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:39.789556    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0f47424-1492-437c-9da2-0712ea4266b5-istiod-ca-cert\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:13:39.795420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:39.789704    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-istio-envoy\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:13:39.795420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:39.789753    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0f47424-1492-437c-9da2-0712ea4266b5-istio-podinfo\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:13:39.795420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:39.789821    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plp92\" (UniqueName: \"kubernetes.io/projected/b0f47424-1492-437c-9da2-0712ea4266b5-kube-api-access-plp92\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:13:39.795420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:39.789863    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-istio-data\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:13:39.795420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:39.789960    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-credential-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:13:39.795420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:39.789985    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0f47424-1492-437c-9da2-0712ea4266b5-istio-token\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:13:39.795420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:39.790030    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-workload-certs\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:13:39.795420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:39.790071    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-workload-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:13:39.808136 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:39.808046    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"]
Oct 31 23:13:39.890972 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:39.890938    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0f47424-1492-437c-9da2-0712ea4266b5-istio-podinfo\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:13:39.891139 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:39.891078    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plp92\" (UniqueName: \"kubernetes.io/projected/b0f47424-1492-437c-9da2-0712ea4266b5-kube-api-access-plp92\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:13:39.891139 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:39.891113    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-istio-data\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:13:39.891222 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:39.891178    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-credential-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:13:39.891222 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:39.891193    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0f47424-1492-437c-9da2-0712ea4266b5-istio-token\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:13:39.891222 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:39.891220    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-workload-certs\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:13:39.891383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:39.891253    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-workload-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:13:39.891383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:39.891343    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0f47424-1492-437c-9da2-0712ea4266b5-istiod-ca-cert\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:13:39.891487 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:39.891469    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-istio-envoy\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:13:39.891600 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:39.891577    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-credential-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:13:39.891642 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:39.891595    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-workload-certs\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:13:39.891694 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:39.891671    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-workload-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:13:39.891875 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:39.891852    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-istio-data\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:13:39.892153 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:39.892136    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0f47424-1492-437c-9da2-0712ea4266b5-istiod-ca-cert\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:13:39.894150 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:39.894126    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-istio-envoy\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:13:39.894333 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:39.894311    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0f47424-1492-437c-9da2-0712ea4266b5-istio-podinfo\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:13:39.902245 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:39.902226    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plp92\" (UniqueName: \"kubernetes.io/projected/b0f47424-1492-437c-9da2-0712ea4266b5-kube-api-access-plp92\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:13:39.902368 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:39.902294    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0f47424-1492-437c-9da2-0712ea4266b5-istio-token\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:13:40.092637 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:40.092556    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:13:40.381095 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:40.381070    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"]
Oct 31 23:13:40.383211 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:13:40.383182    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f47424_1492_437c_9da2_0712ea4266b5.slice/crio-e5a527a5dcfcf8acb78ce13059e226554f02dfb340abdebede69501683766342 WatchSource:0}: Error finding container e5a527a5dcfcf8acb78ce13059e226554f02dfb340abdebede69501683766342: Status 404 returned error can't find the container with id e5a527a5dcfcf8acb78ce13059e226554f02dfb340abdebede69501683766342
Oct 31 23:13:40.386032 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:40.385998    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:13:40.386111 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:40.386067    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:13:40.386111 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:40.386096    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:13:41.058662 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:41.058624    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz" event={"ID":"b0f47424-1492-437c-9da2-0712ea4266b5","Type":"ContainerStarted","Data":"b3d2e931eb8f500c51cb822f0538e89808c092ca3f05650b780f8554704d49d0"}
Oct 31 23:13:41.059133 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:41.058706    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz" event={"ID":"b0f47424-1492-437c-9da2-0712ea4266b5","Type":"ContainerStarted","Data":"e5a527a5dcfcf8acb78ce13059e226554f02dfb340abdebede69501683766342"}
Oct 31 23:13:41.080719 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:41.080662    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz" podStartSLOduration=2.080647454 podStartE2EDuration="2.080647454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:13:41.079070391 +0000 UTC m=+1439.826259891" watchObservedRunningTime="2026-10-31 23:13:41.080647454 +0000 UTC m=+1439.827836961"
Oct 31 23:13:41.093481 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:41.093453    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:13:41.094748 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:41.094724    4301 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 23:13:41.094863 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:41.094768    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz" podUID="b0f47424-1492-437c-9da2-0712ea4266b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 23:13:42.093768 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:42.093734    4301 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 23:13:42.094169 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:42.093792    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz" podUID="b0f47424-1492-437c-9da2-0712ea4266b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 23:13:43.098398 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:43.096334    4301 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 23:13:43.098398 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:43.096440    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz" podUID="b0f47424-1492-437c-9da2-0712ea4266b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 23:13:44.093224 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:44.093189    4301 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 23:13:44.093443 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:44.093243    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz" podUID="b0f47424-1492-437c-9da2-0712ea4266b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 23:13:45.093852 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:45.093819    4301 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 23:13:45.094255 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:45.093871    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz" podUID="b0f47424-1492-437c-9da2-0712ea4266b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 23:13:46.093001 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:46.092966    4301 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 23:13:46.093181 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:46.093018    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz" podUID="b0f47424-1492-437c-9da2-0712ea4266b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 23:13:47.095379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:47.094900    4301 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 23:13:47.095379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:47.094973    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz" podUID="b0f47424-1492-437c-9da2-0712ea4266b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 23:13:47.885622 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:47.884474    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:48.025578 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:48.025529    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:48.048753 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:48.047499    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:48.097071 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:48.096500    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:48.097983 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:48.097956    4301 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 23:13:48.098630 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:48.098603    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz" podUID="b0f47424-1492-437c-9da2-0712ea4266b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 23:13:48.134699 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:48.134656    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:48.254563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:48.254460    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:48.536462 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:48.521510    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:48.550408 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:48.542491    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:48.583383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:48.580880    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 23:13:48.830432 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:48.829412    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:48.869385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:48.863694    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 31 23:13:48.917382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:48.885453    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:48.917382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:48.885945    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:13:49.048416 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:49.047871    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:49.072385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:49.048637    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:13:49.152338 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:49.152299    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:49.157506 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:49.157480    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:13:49.201010 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:49.176756    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:49.278234 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:49.278184    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:49.317961 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:49.283528    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:13:49.318281 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:49.318261    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:13:49.319690 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:49.285761    4301 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 23:13:49.319861 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:49.319838    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz" podUID="b0f47424-1492-437c-9da2-0712ea4266b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 23:13:49.722384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:49.699530    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 23:13:49.750682 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:49.750614    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:49.963822 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:49.963782    4301 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 23:13:49.984697 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:49.984595    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq" podUID="cdcadd41-cef3-4fbc-9551-ed8dae06f790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:50.034661 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:50.034611    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:50.153639 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:50.153574    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:50.197601 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:50.184007    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:50.197601 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:50.192268    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:50.197601 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:50.195948    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:50.208554 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:50.208510    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:50.339645 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:50.339542    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 23:13:50.365096 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:50.365062    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:13:50.365426 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:50.365409    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:13:50.365629 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:50.365596    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:13:50.369510 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:50.369487    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:13:50.373484 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:50.373464    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:13:50.400150 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:50.398923    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:13:50.703636 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:50.703580    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded"
Oct 31 23:13:50.704128 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:50.704099    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:50.747707 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:50.747666    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:50.775419 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:50.768672    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:51.062606 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:51.062516    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:13:51.161388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:51.159555    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:51.163510 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:51.163137    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 23:13:51.212015 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:51.187521    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:51.230495 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:51.230458    4301 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 23:13:51.230911 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:51.230880    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz" podUID="b0f47424-1492-437c-9da2-0712ea4266b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:51.258152 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:51.211444    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 31 23:13:51.258776 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:51.211976    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:51.316638 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:51.316531    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:51.326780 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:51.326749    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:13:51.331674 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:51.330770    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:13:51.339982 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:51.339942    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:51.707726 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:51.707683    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:51.773384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:51.749211    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:51.773982 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:51.770436    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:51.800804 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:51.774587    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:13:52.242382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:52.216511    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:52.288199 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:52.230476    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 23:13:52.352386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:52.254634    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:52.358675 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:52.358639    4301 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 23:13:52.358960 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:52.358926    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz" podUID="b0f47424-1492-437c-9da2-0712ea4266b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:52.359445 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:52.359091    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:52.482426 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:52.478623    4301 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 23:13:52.482426 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:52.478690    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs" podUID="944451b3-0e62-4419-bf3a-37401268d08f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:52.710276 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:52.709699    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:52.775382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:52.774466    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:52.807835 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:52.807788    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:53.102574 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:53.102465    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:53.399168 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:53.385739    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:53.427389 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:53.400872    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:53.427389 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:53.425333    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:53.452386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:53.439242    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 31 23:13:53.452386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:53.448099    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:53.452386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:53.450176    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:53.490840 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:53.490802    4301 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 23:13:53.491180 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:53.491128    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz" podUID="b0f47424-1492-437c-9da2-0712ea4266b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:53.573384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:53.565885    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:53.773502 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:53.731339    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:53.836050 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:53.776578    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:53.953857 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:53.952276    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:53.953857 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:53.953146    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": context deadline exceeded"
Oct 31 23:13:54.028452 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:54.022747    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:13:54.030180 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:54.029223    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:13:54.172545 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:54.121071    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:13:54.222913 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:54.222868    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:54.223777 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:54.223753    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:13:54.258731 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:54.249121    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:13:54.418106 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:54.418076    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:13:54.420121 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:54.420087    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:13:54.445789 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:54.436489    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:54.445789 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:54.441405    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 31 23:13:54.445789 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:54.441789    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:13:54.465667 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:54.463700    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 23:13:54.465667 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:54.465495    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 31 23:13:54.469443 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:54.469415    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:13:54.480120 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:54.469872    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:13:54.553762 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:54.553735    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:13:54.573508 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:54.573455    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:13:54.611713 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:54.611685    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:13:54.611899 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:54.611737    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:13:54.611899 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:54.611770    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:13:54.612036 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:54.611949    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:13:54.615667 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:54.615641    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:13:58.707217 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:58.707178    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe640623-tvcgh"]
Oct 31 23:13:58.712565 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:58.712539    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe640623-tvcgh"
Oct 31 23:13:58.716802 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:58.716775    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\""
Oct 31 23:13:58.716966 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:58.716887    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 23:13:58.721727 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:58.721311    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe640623-tvcgh"]
Oct 31 23:13:58.760950 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:58.760912    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmwhr\" (UniqueName: \"kubernetes.io/projected/13bf0fc3-47ae-4bb3-8326-6a3e059d47da-kube-api-access-qmwhr\") pod \"maas-api-revoke-sub-fe640623-tvcgh\" (UID: \"13bf0fc3-47ae-4bb3-8326-6a3e059d47da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe640623-tvcgh"
Oct 31 23:13:58.761157 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:58.760975    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13bf0fc3-47ae-4bb3-8326-6a3e059d47da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe640623-tvcgh\" (UID: \"13bf0fc3-47ae-4bb3-8326-6a3e059d47da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe640623-tvcgh"
Oct 31 23:13:58.761157 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:58.761100    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13bf0fc3-47ae-4bb3-8326-6a3e059d47da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe640623-tvcgh\" (UID: \"13bf0fc3-47ae-4bb3-8326-6a3e059d47da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe640623-tvcgh"
Oct 31 23:13:58.862946 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:58.862901    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13bf0fc3-47ae-4bb3-8326-6a3e059d47da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe640623-tvcgh\" (UID: \"13bf0fc3-47ae-4bb3-8326-6a3e059d47da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe640623-tvcgh"
Oct 31 23:13:58.863162 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:58.863147    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmwhr\" (UniqueName: \"kubernetes.io/projected/13bf0fc3-47ae-4bb3-8326-6a3e059d47da-kube-api-access-qmwhr\") pod \"maas-api-revoke-sub-fe640623-tvcgh\" (UID: \"13bf0fc3-47ae-4bb3-8326-6a3e059d47da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe640623-tvcgh"
Oct 31 23:13:58.863217 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:58.863182    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13bf0fc3-47ae-4bb3-8326-6a3e059d47da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe640623-tvcgh\" (UID: \"13bf0fc3-47ae-4bb3-8326-6a3e059d47da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe640623-tvcgh"
Oct 31 23:13:58.863603 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:58.863579    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13bf0fc3-47ae-4bb3-8326-6a3e059d47da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe640623-tvcgh\" (UID: \"13bf0fc3-47ae-4bb3-8326-6a3e059d47da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe640623-tvcgh"
Oct 31 23:13:58.865655 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:58.865634    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13bf0fc3-47ae-4bb3-8326-6a3e059d47da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe640623-tvcgh\" (UID: \"13bf0fc3-47ae-4bb3-8326-6a3e059d47da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe640623-tvcgh"
Oct 31 23:13:58.878838 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:58.878811    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmwhr\" (UniqueName: \"kubernetes.io/projected/13bf0fc3-47ae-4bb3-8326-6a3e059d47da-kube-api-access-qmwhr\") pod \"maas-api-revoke-sub-fe640623-tvcgh\" (UID: \"13bf0fc3-47ae-4bb3-8326-6a3e059d47da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe640623-tvcgh"
Oct 31 23:13:59.029506 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:59.028728    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe640623-tvcgh"
Oct 31 23:13:59.322208 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:59.322119    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe640623-tvcgh"]
Oct 31 23:13:59.327965 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:13:59.327939    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13bf0fc3_47ae_4bb3_8326_6a3e059d47da.slice/crio-b55fd87af87769c3c6820a550c97340f73df0c6b18579595100368894423c6c1 WatchSource:0}: Error finding container b55fd87af87769c3c6820a550c97340f73df0c6b18579595100368894423c6c1: Status 404 returned error can't find the container with id b55fd87af87769c3c6820a550c97340f73df0c6b18579595100368894423c6c1
Oct 31 23:13:59.736599 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:13:59.736557    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe640623-tvcgh" event={"ID":"13bf0fc3-47ae-4bb3-8326-6a3e059d47da","Type":"ContainerStarted","Data":"b55fd87af87769c3c6820a550c97340f73df0c6b18579595100368894423c6c1"}
Oct 31 23:14:00.743972 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:00.743939    4301 generic.go:356] "Generic (PLEG): container finished" podID="13bf0fc3-47ae-4bb3-8326-6a3e059d47da" containerID="e314a5658651f8fb7824743383a36a3cfe04b9825295619b07e0c2f932a6cc08" exitCode=0
Oct 31 23:14:00.744415 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:00.744016    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe640623-tvcgh" event={"ID":"13bf0fc3-47ae-4bb3-8326-6a3e059d47da","Type":"ContainerDied","Data":"e314a5658651f8fb7824743383a36a3cfe04b9825295619b07e0c2f932a6cc08"}
Oct 31 23:14:01.972288 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:01.972261    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe640623-tvcgh"
Oct 31 23:14:02.030065 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:02.029957    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13bf0fc3-47ae-4bb3-8326-6a3e059d47da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13bf0fc3-47ae-4bb3-8326-6a3e059d47da-maas-api-openshift-ca\") pod \"13bf0fc3-47ae-4bb3-8326-6a3e059d47da\" (UID: \"13bf0fc3-47ae-4bb3-8326-6a3e059d47da\") "
Oct 31 23:14:02.030065 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:02.030040    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13bf0fc3-47ae-4bb3-8326-6a3e059d47da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13bf0fc3-47ae-4bb3-8326-6a3e059d47da-maas-api-xks-ca\") pod \"13bf0fc3-47ae-4bb3-8326-6a3e059d47da\" (UID: \"13bf0fc3-47ae-4bb3-8326-6a3e059d47da\") "
Oct 31 23:14:02.030341 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:02.030322    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13bf0fc3-47ae-4bb3-8326-6a3e059d47da-kube-api-access-qmwhr\" (UniqueName: \"kubernetes.io/projected/13bf0fc3-47ae-4bb3-8326-6a3e059d47da-kube-api-access-qmwhr\") pod \"13bf0fc3-47ae-4bb3-8326-6a3e059d47da\" (UID: \"13bf0fc3-47ae-4bb3-8326-6a3e059d47da\") "
Oct 31 23:14:02.030819 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:02.030788    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13bf0fc3-47ae-4bb3-8326-6a3e059d47da-maas-api-openshift-ca" pod "13bf0fc3-47ae-4bb3-8326-6a3e059d47da" (UID: "13bf0fc3-47ae-4bb3-8326-6a3e059d47da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:14:02.031019 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:02.030997    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13bf0fc3-47ae-4bb3-8326-6a3e059d47da-maas-api-openshift-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:14:02.032643 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:02.032512    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13bf0fc3-47ae-4bb3-8326-6a3e059d47da-maas-api-xks-ca" pod "13bf0fc3-47ae-4bb3-8326-6a3e059d47da" (UID: "13bf0fc3-47ae-4bb3-8326-6a3e059d47da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:14:02.040857 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:02.040833    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13bf0fc3-47ae-4bb3-8326-6a3e059d47da-kube-api-access-qmwhr" pod "13bf0fc3-47ae-4bb3-8326-6a3e059d47da" (UID: "13bf0fc3-47ae-4bb3-8326-6a3e059d47da"). InnerVolumeSpecName "kube-api-access-qmwhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:14:02.132551 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:02.132519    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmwhr\" (UniqueName: \"kubernetes.io/projected/13bf0fc3-47ae-4bb3-8326-6a3e059d47da-kube-api-access-qmwhr\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:14:02.132551 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:02.132547    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13bf0fc3-47ae-4bb3-8326-6a3e059d47da-maas-api-xks-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:14:02.628662 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:14:02.628623    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:14:02.765088 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:02.765058    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe640623-tvcgh"
Oct 31 23:14:02.765254 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:02.765051    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe640623-tvcgh" event={"ID":"13bf0fc3-47ae-4bb3-8326-6a3e059d47da","Type":"ContainerDied","Data":"b55fd87af87769c3c6820a550c97340f73df0c6b18579595100368894423c6c1"}
Oct 31 23:14:02.765254 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:02.765180    4301 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b55fd87af87769c3c6820a550c97340f73df0c6b18579595100368894423c6c1"
Oct 31 23:14:03.636919 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:03.636887    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:14:05.759382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:05.758459    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:05.912390 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:05.861483    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 23:14:06.187077 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:06.181048    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:06.187077 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:06.182224    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:06.213594 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:06.204731    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:06.385391 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:06.369346    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 23:14:06.421632 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:06.421568    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:06.619447 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:06.574658    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:06.619447 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:06.604511    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:06.759611 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:06.759567    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:06.763119 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:06.763093    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:14:06.830596 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:06.830553    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:06.892483 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:06.891546    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:06.937556 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:06.937508    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 23:14:06.948417 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:06.938146    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:14:07.327531 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:07.327431    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 23:14:07.349313 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:07.330473    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:07.349939 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:07.349914    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:14:07.350116 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:07.350103    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:14:07.386383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:07.330517    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:07.494441 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:07.386657    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded"
Oct 31 23:14:07.494697 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:07.400904    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 23:14:07.511520 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:07.400952    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Oct 31 23:14:07.511775 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:07.427344    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:07.576599 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:07.576548    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 31 23:14:07.609613 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:07.594460    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:07.624076 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:07.619444    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:07.770401 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:07.762831    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:14:07.770401 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:07.764656    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:14:07.770401 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:07.765982    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:14:07.770401 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:07.766034    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:14:07.770401 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:07.766071    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:14:07.770401 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:07.767343    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 23:14:07.925540 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:07.925481    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:07.946381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:07.926422    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:14:07.947537 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:07.943586    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:08.026384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:08.022576    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:08.153379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:08.148276    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:14:08.354426 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:08.326992    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:14:08.354426 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:08.354110    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 31 23:14:08.356018 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:08.355415    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 23:14:08.475876 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:08.475840    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:14:08.503384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:08.500904    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:08.503384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:08.502263    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 23:14:08.708890 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:08.708838    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 31 23:14:08.819385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:08.811760    4301 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 23:14:08.819385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:08.811823    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq" podUID="cc182c32-7804-4e3f-b6eb-d22418055d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:08.819385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:08.815587    4301 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 23:14:08.819385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:08.815640    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz" podUID="3b0d47aa-36ab-46fa-b41f-20a71f2256cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:08.934384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:08.933454    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:09.028469 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:08.987823    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 23:14:09.369445 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:09.357603    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 23:14:09.369445 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:09.358318    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 31 23:14:09.369445 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:09.360005    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 31 23:14:09.468558 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:09.467684    4301 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 23:14:09.468558 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:09.467742    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g" podUID="85e9dd14-aa39-400e-b672-4c3122ea568a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:09.521385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:09.506516    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 31 23:14:09.521385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:09.507681    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 23:14:09.521385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:09.507964    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 23:14:09.750561 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:09.750446    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 31 23:14:09.826388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:09.769468    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:09.938770 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:09.938726    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 23:14:10.139431 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:10.037861    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 23:14:10.139431 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:10.079464    4301 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 23:14:10.139431 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:10.079528    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz" podUID="b0f47424-1492-437c-9da2-0712ea4266b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:10.395922 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:10.395819    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:10.423567 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:10.423512    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:14:10.424570 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:10.398524    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 23:14:10.424570 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:10.423080    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:10.494460 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:10.494419    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"]
Oct 31 23:14:10.536391 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:10.523036    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:10.536391 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:10.527485    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 23:14:10.538082 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:10.538049    4301 cpu_manager.go:414] "containerMap: removing container" podUID="13bf0fc3-47ae-4bb3-8326-6a3e059d47da" containerName="revoke-keys"
Oct 31 23:14:10.538271 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:10.538255    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13bf0fc3-47ae-4bb3-8326-6a3e059d47da" containerName="revoke-keys"
Oct 31 23:14:10.538537 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:10.538520    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="13bf0fc3-47ae-4bb3-8326-6a3e059d47da" containerName="revoke-keys"
Oct 31 23:14:10.588388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:10.562724    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:10.588388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:10.585643    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 31 23:14:10.698657 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:10.698571    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"]
Oct 31 23:14:10.698948 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:10.698929    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:14:10.700047 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:10.700024    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:14:10.733576 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:10.733537    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istio-podinfo\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:14:10.733964 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:10.733942    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzpvt\" (UniqueName: \"kubernetes.io/projected/bf839f5e-2570-44bd-9c62-7a65ea1590ed-kube-api-access-wzpvt\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:14:10.761398 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:10.734217    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istiod-ca-cert\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:14:10.763959 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:10.763929    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-credential-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:14:10.797477 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:10.797431    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-workload-certs\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:14:10.797974 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:10.797811    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istio-data\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:14:10.798268 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:10.798158    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istio-token\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:14:10.798944 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:10.798923    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istio-envoy\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:14:10.836433 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:10.827488    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-workload-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:14:10.836433 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:10.815764    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 31 23:14:10.836433 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:10.815723    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:10.836433 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:10.828783    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:14:10.951525 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:10.950324    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:11.018399 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:10.951656    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzpvt\" (UniqueName: \"kubernetes.io/projected/bf839f5e-2570-44bd-9c62-7a65ea1590ed-kube-api-access-wzpvt\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:14:11.018399 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:11.016846    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istiod-ca-cert\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:14:11.018399 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:11.016905    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-credential-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:14:11.018399 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:11.017016    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-workload-certs\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:14:11.018399 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:11.017046    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istio-data\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:14:11.018399 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:11.017120    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istio-token\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:14:11.018399 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:11.017352    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istio-envoy\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:14:11.018399 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:11.017409    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-workload-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:14:11.018399 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:11.017528    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istio-podinfo\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:14:11.042074 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:11.042020    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-workload-certs\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:14:11.049438 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:11.048324    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istio-data\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:14:11.053445 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:11.049891    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istiod-ca-cert\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:14:11.057440 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:11.054400    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-workload-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:14:11.095793 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:11.067811    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:11.114799 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:11.065528    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:11.185172 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:11.076998    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-credential-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:14:11.189467 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:11.176094    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzpvt\" (UniqueName: \"kubernetes.io/projected/bf839f5e-2570-44bd-9c62-7a65ea1590ed-kube-api-access-wzpvt\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:14:11.189691 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:11.183021    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istio-envoy\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:14:11.216450 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:11.214350    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istio-token\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:14:11.220386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:11.217488    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istio-podinfo\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:14:11.440392 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:11.436974    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:11.501739 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:11.501653    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:14:11.504523 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:11.504485    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:11.814556 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:11.504735    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:11.818380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:11.535514    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 23:14:11.823288 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:11.535564    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:11.824578 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:11.596566    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:11.858276 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:11.842563    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:11.858276 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:11.842894    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 31 23:14:12.042383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:12.025132    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 23:14:12.206510 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:12.205371    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:12.363722 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:12.359013    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:14:12.398435 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:12.394626    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:14:12.444487 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:12.444462    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:14:12.445021 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:12.444993    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:12.480962 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:12.480890    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:14:12.484734 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:12.484711    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:14:12.541601 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:12.523773    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:14:12.544046 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:12.544014    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:14:12.551805 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:12.551737    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:14:12.562219 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:12.552557    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:14:12.585906 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:12.576900    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:14:12.674403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:12.674068    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"]
Oct 31 23:14:12.678749 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:14:12.678720    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf839f5e_2570_44bd_9c62_7a65ea1590ed.slice/crio-2466c6f4b9280464d9cadb5d61823f7c3347f2e4d753bb01ef039172b8cd97f8 WatchSource:0}: Error finding container 2466c6f4b9280464d9cadb5d61823f7c3347f2e4d753bb01ef039172b8cd97f8: Status 404 returned error can't find the container with id 2466c6f4b9280464d9cadb5d61823f7c3347f2e4d753bb01ef039172b8cd97f8
Oct 31 23:14:12.682153 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:12.682111    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:14:12.682266 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:12.682200    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:14:12.682266 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:12.682245    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:14:13.359775 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:13.359727    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2" event={"ID":"bf839f5e-2570-44bd-9c62-7a65ea1590ed","Type":"ContainerStarted","Data":"4706b538ebf255543d9badb25c61d0a5dc8efc72762b1e2dd71a58e52bc31c1e"}
Oct 31 23:14:13.360198 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:13.359792    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2" event={"ID":"bf839f5e-2570-44bd-9c62-7a65ea1590ed","Type":"ContainerStarted","Data":"2466c6f4b9280464d9cadb5d61823f7c3347f2e4d753bb01ef039172b8cd97f8"}
Oct 31 23:14:13.384209 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:13.384161    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2" podStartSLOduration=3.384147588 podStartE2EDuration="3.384147588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:14:13.383444893 +0000 UTC m=+1472.130634434" watchObservedRunningTime="2026-10-31 23:14:13.384147588 +0000 UTC m=+1472.131337092"
Oct 31 23:14:13.505613 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:13.505582    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:14:13.507047 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:13.507019    4301 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 23:14:13.507207 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:13.507061    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2" podUID="bf839f5e-2570-44bd-9c62-7a65ea1590ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 23:14:14.505157 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:14.505123    4301 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 23:14:14.505670 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:14.505176    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2" podUID="bf839f5e-2570-44bd-9c62-7a65ea1590ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 23:14:15.505121 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:15.505085    4301 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 23:14:15.505321 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:15.505155    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2" podUID="bf839f5e-2570-44bd-9c62-7a65ea1590ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 23:14:16.505518 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:16.505472    4301 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 23:14:16.506010 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:16.505557    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2" podUID="bf839f5e-2570-44bd-9c62-7a65ea1590ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 23:14:17.352662 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:17.352622    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10903-578db2"]
Oct 31 23:14:17.362293 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:17.362265    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10903-578db2"
Oct 31 23:14:17.363835 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:17.363774    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10903-578db2"]
Oct 31 23:14:17.367576 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:17.366777    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-rq76x\""
Oct 31 23:14:17.425033 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:17.424996    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd2tb\" (UniqueName: \"kubernetes.io/projected/a587bcee-0f91-41f9-beaa-b2de850c9119-kube-api-access-pd2tb\") pod \"test-curl-10903-578db2\" (UID: \"a587bcee-0f91-41f9-beaa-b2de850c9119\") " pod="opendatahub/test-curl-10903-578db2"
Oct 31 23:14:17.504766 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:17.504722    4301 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 23:14:17.504952 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:17.504780    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2" podUID="bf839f5e-2570-44bd-9c62-7a65ea1590ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 23:14:17.529379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:17.527172    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd2tb\" (UniqueName: \"kubernetes.io/projected/a587bcee-0f91-41f9-beaa-b2de850c9119-kube-api-access-pd2tb\") pod \"test-curl-10903-578db2\" (UID: \"a587bcee-0f91-41f9-beaa-b2de850c9119\") " pod="opendatahub/test-curl-10903-578db2"
Oct 31 23:14:17.570380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:17.567263    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd2tb\" (UniqueName: \"kubernetes.io/projected/a587bcee-0f91-41f9-beaa-b2de850c9119-kube-api-access-pd2tb\") pod \"test-curl-10903-578db2\" (UID: \"a587bcee-0f91-41f9-beaa-b2de850c9119\") " pod="opendatahub/test-curl-10903-578db2"
Oct 31 23:14:17.689379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:17.688983    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10903-578db2"
Oct 31 23:14:17.994903 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:17.994878    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10903-578db2"]
Oct 31 23:14:17.996669 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:14:17.996641    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda587bcee_0f91_41f9_beaa_b2de850c9119.slice/crio-246995339eb57e1cb444e9c87693eba3e1daa116a6cba957dc3f8ba7039a0e6d WatchSource:0}: Error finding container 246995339eb57e1cb444e9c87693eba3e1daa116a6cba957dc3f8ba7039a0e6d: Status 404 returned error can't find the container with id 246995339eb57e1cb444e9c87693eba3e1daa116a6cba957dc3f8ba7039a0e6d
Oct 31 23:14:18.465418 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:18.465381    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10903-578db2" event={"ID":"a587bcee-0f91-41f9-beaa-b2de850c9119","Type":"ContainerStarted","Data":"246995339eb57e1cb444e9c87693eba3e1daa116a6cba957dc3f8ba7039a0e6d"}
Oct 31 23:14:18.505460 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:18.505426    4301 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 23:14:18.505647 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:18.505517    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2" podUID="bf839f5e-2570-44bd-9c62-7a65ea1590ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 23:14:19.473841 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:19.473795    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10903-578db2" event={"ID":"a587bcee-0f91-41f9-beaa-b2de850c9119","Type":"ContainerStarted","Data":"5150017d9bcf0d6135e1fb9804c1e81b36f60f832d552a76f407606ebebb9abd"}
Oct 31 23:14:19.491680 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:19.491630    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-10903-578db2" podStartSLOduration=1.450334285 podStartE2EDuration="2.491615945s" totalImagesPullingTime="1.04128166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:14:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:14:19.487391513 +0000 UTC m=+1478.234581008" watchObservedRunningTime="2026-10-31 23:14:19.491615945 +0000 UTC m=+1478.238805468"
Oct 31 23:14:19.505178 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:19.505149    4301 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 23:14:19.505343 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:19.505203    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2" podUID="bf839f5e-2570-44bd-9c62-7a65ea1590ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 23:14:20.232587 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:20.232548    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10903-578db2"]
Oct 31 23:14:20.237009 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:20.236979    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10903-578db2"]
Oct 31 23:14:20.407688 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:20.407654    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10903-064925"]
Oct 31 23:14:20.408602 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:20.408582    4301 cpu_manager.go:414] "containerMap: removing container" podUID="a587bcee-0f91-41f9-beaa-b2de850c9119" containerName="test-curl-10903-578db2"
Oct 31 23:14:20.408674 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:20.408606    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a587bcee-0f91-41f9-beaa-b2de850c9119" containerName="test-curl-10903-578db2"
Oct 31 23:14:20.408744 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:20.408732    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="a587bcee-0f91-41f9-beaa-b2de850c9119" containerName="test-curl-10903-578db2"
Oct 31 23:14:20.412684 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:20.412665    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10903-064925"
Oct 31 23:14:20.422020 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:20.421996    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10903-064925"]
Oct 31 23:14:20.553443 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:20.536475    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fwm5\" (UniqueName: \"kubernetes.io/projected/913b2d55-ba28-40c3-b19f-de5ee21ffe84-kube-api-access-8fwm5\") pod \"test-curl-10903-064925\" (UID: \"913b2d55-ba28-40c3-b19f-de5ee21ffe84\") " pod="opendatahub/test-curl-10903-064925"
Oct 31 23:14:20.697384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:20.678468    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fwm5\" (UniqueName: \"kubernetes.io/projected/913b2d55-ba28-40c3-b19f-de5ee21ffe84-kube-api-access-8fwm5\") pod \"test-curl-10903-064925\" (UID: \"913b2d55-ba28-40c3-b19f-de5ee21ffe84\") " pod="opendatahub/test-curl-10903-064925"
Oct 31 23:14:20.706777 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:20.706747    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:14:20.719104 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:20.718238    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:14:20.719636 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:20.719612    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:14:20.777051 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:20.777014    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fwm5\" (UniqueName: \"kubernetes.io/projected/913b2d55-ba28-40c3-b19f-de5ee21ffe84-kube-api-access-8fwm5\") pod \"test-curl-10903-064925\" (UID: \"913b2d55-ba28-40c3-b19f-de5ee21ffe84\") " pod="opendatahub/test-curl-10903-064925"
Oct 31 23:14:21.042639 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:21.042607    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10903-064925"
Oct 31 23:14:21.326666 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:21.326639    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10903-064925"]
Oct 31 23:14:21.328353 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:14:21.328329    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod913b2d55_ba28_40c3_b19f_de5ee21ffe84.slice/crio-20c0c21e9ce353c658d32d2fe097cb92cd9cfbfbcffaff7dcd09a5d1db8a8201 WatchSource:0}: Error finding container 20c0c21e9ce353c658d32d2fe097cb92cd9cfbfbcffaff7dcd09a5d1db8a8201: Status 404 returned error can't find the container with id 20c0c21e9ce353c658d32d2fe097cb92cd9cfbfbcffaff7dcd09a5d1db8a8201
Oct 31 23:14:21.696701 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:21.696658    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10903-064925" event={"ID":"913b2d55-ba28-40c3-b19f-de5ee21ffe84","Type":"ContainerStarted","Data":"634ed05b296dee7ff8dd57a00e2ef566ca235bd70c9861e47891c9208fac6f82"}
Oct 31 23:14:21.697127 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:21.696709    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10903-064925" event={"ID":"913b2d55-ba28-40c3-b19f-de5ee21ffe84","Type":"ContainerStarted","Data":"20c0c21e9ce353c658d32d2fe097cb92cd9cfbfbcffaff7dcd09a5d1db8a8201"}
Oct 31 23:14:21.697127 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:21.696770    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-10903-578db2" podUID="a587bcee-0f91-41f9-beaa-b2de850c9119" containerName="test-curl-10903-578db2" containerID="cri-o://5150017d9bcf0d6135e1fb9804c1e81b36f60f832d552a76f407606ebebb9abd" gracePeriod=2
Oct 31 23:14:21.713726 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:21.713679    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-10903-064925" podStartSLOduration=1.713665428 podStartE2EDuration="1.713665428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:14:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:14:21.710140424 +0000 UTC m=+1480.457329929" watchObservedRunningTime="2026-10-31 23:14:21.713665428 +0000 UTC m=+1480.460854932"
Oct 31 23:14:22.561738 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:22.561706    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10903-064925"]
Oct 31 23:14:22.573774 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:22.573745    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10903-064925"]
Oct 31 23:14:23.708768 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:23.708733    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-10903-064925" podUID="913b2d55-ba28-40c3-b19f-de5ee21ffe84" containerName="test-curl-10903-064925" containerID="cri-o://634ed05b296dee7ff8dd57a00e2ef566ca235bd70c9861e47891c9208fac6f82" gracePeriod=2
Oct 31 23:14:24.006967 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.006477    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10903-578db2"
Oct 31 23:14:24.078504 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.078462    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a587bcee-0f91-41f9-beaa-b2de850c9119-kube-api-access-pd2tb\" (UniqueName: \"kubernetes.io/projected/a587bcee-0f91-41f9-beaa-b2de850c9119-kube-api-access-pd2tb\") pod \"a587bcee-0f91-41f9-beaa-b2de850c9119\" (UID: \"a587bcee-0f91-41f9-beaa-b2de850c9119\") "
Oct 31 23:14:24.080768 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.080736    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a587bcee-0f91-41f9-beaa-b2de850c9119-kube-api-access-pd2tb" pod "a587bcee-0f91-41f9-beaa-b2de850c9119" (UID: "a587bcee-0f91-41f9-beaa-b2de850c9119"). InnerVolumeSpecName "kube-api-access-pd2tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:14:24.124598 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.124564    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"]
Oct 31 23:14:24.125419 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.125395    4301 cpu_manager.go:414] "containerMap: removing container" podUID="913b2d55-ba28-40c3-b19f-de5ee21ffe84" containerName="test-curl-10903-064925"
Oct 31 23:14:24.125419 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.125418    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="913b2d55-ba28-40c3-b19f-de5ee21ffe84" containerName="test-curl-10903-064925"
Oct 31 23:14:24.125598 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.125560    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="913b2d55-ba28-40c3-b19f-de5ee21ffe84" containerName="test-curl-10903-064925"
Oct 31 23:14:24.129526 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.129495    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:14:24.144330 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.143477    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-ed6117ae-gw-openshift-default-dockercfg-jqfql\""
Oct 31 23:14:24.150155 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.150125    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"]
Oct 31 23:14:24.180419 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.180315    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-token\") pod \"e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") " pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:14:24.180597 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.180450    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-data\") pod \"e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") " pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:14:24.180597 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.180514    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-workload-certs\") pod \"e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") " pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:14:24.180735 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.180708    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-credential-socket\") pod \"e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") " pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:14:24.180843 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.180824    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-envoy\") pod \"e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") " pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:14:24.180922 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.180896    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjqcz\" (UniqueName: \"kubernetes.io/projected/3f037628-d51c-48e0-be4e-5d9c99a10332-kube-api-access-pjqcz\") pod \"e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") " pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:14:24.181012 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.180996    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f037628-d51c-48e0-be4e-5d9c99a10332-istiod-ca-cert\") pod \"e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") " pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:14:24.181102 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.181085    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-podinfo\") pod \"e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") " pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:14:24.181229 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.181213    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-workload-socket\") pod \"e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") " pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:14:24.181337 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.181324    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd2tb\" (UniqueName: \"kubernetes.io/projected/a587bcee-0f91-41f9-beaa-b2de850c9119-kube-api-access-pd2tb\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:14:24.282805 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.282718    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjqcz\" (UniqueName: \"kubernetes.io/projected/3f037628-d51c-48e0-be4e-5d9c99a10332-kube-api-access-pjqcz\") pod \"e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") " pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:14:24.282805 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.282787    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f037628-d51c-48e0-be4e-5d9c99a10332-istiod-ca-cert\") pod \"e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") " pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:14:24.283051 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.282830    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-podinfo\") pod \"e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") " pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:14:24.283051 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.282914    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-workload-socket\") pod \"e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") " pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:14:24.283051 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.282933    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-token\") pod \"e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") " pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:14:24.283051 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.282998    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-data\") pod \"e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") " pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:14:24.283051 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.283045    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-workload-certs\") pod \"e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") " pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:14:24.283312 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.283100    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-credential-socket\") pod \"e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") " pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:14:24.283312 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.283166    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-envoy\") pod \"e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") " pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:14:24.283443 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.283352    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-workload-socket\") pod \"e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") " pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:14:24.283502 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.283450    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-data\") pod \"e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") " pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:14:24.283613 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.283588    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-workload-certs\") pod \"e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") " pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:14:24.283674 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.283632    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-credential-socket\") pod \"e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") " pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:14:24.283770 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.283745    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f037628-d51c-48e0-be4e-5d9c99a10332-istiod-ca-cert\") pod \"e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") " pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:14:24.285473 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.285452    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-envoy\") pod \"e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") " pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:14:24.285549 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.285487    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-podinfo\") pod \"e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") " pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:14:24.297713 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.297686    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjqcz\" (UniqueName: \"kubernetes.io/projected/3f037628-d51c-48e0-be4e-5d9c99a10332-kube-api-access-pjqcz\") pod \"e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") " pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:14:24.299293 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.299267    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-token\") pod \"e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") " pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:14:24.464394 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.460954    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:14:24.836597 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.824007    4301 generic.go:356] "Generic (PLEG): container finished" podID="a587bcee-0f91-41f9-beaa-b2de850c9119" containerID="5150017d9bcf0d6135e1fb9804c1e81b36f60f832d552a76f407606ebebb9abd" exitCode=137
Oct 31 23:14:24.836597 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.824248    4301 scope.go:122] "RemoveContainer" containerID="5150017d9bcf0d6135e1fb9804c1e81b36f60f832d552a76f407606ebebb9abd"
Oct 31 23:14:24.836597 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:24.824495    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10903-578db2"
Oct 31 23:14:25.067818 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:25.062048    4301 scope.go:122] "RemoveContainer" containerID="5150017d9bcf0d6135e1fb9804c1e81b36f60f832d552a76f407606ebebb9abd"
Oct 31 23:14:25.084915 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:14:25.071022    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5150017d9bcf0d6135e1fb9804c1e81b36f60f832d552a76f407606ebebb9abd\": container with ID starting with 5150017d9bcf0d6135e1fb9804c1e81b36f60f832d552a76f407606ebebb9abd not found: ID does not exist" containerID="5150017d9bcf0d6135e1fb9804c1e81b36f60f832d552a76f407606ebebb9abd"
Oct 31 23:14:25.084915 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:25.071065    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5150017d9bcf0d6135e1fb9804c1e81b36f60f832d552a76f407606ebebb9abd"} err="failed to get container status \"5150017d9bcf0d6135e1fb9804c1e81b36f60f832d552a76f407606ebebb9abd\": rpc error: code = NotFound desc = could not find container \"5150017d9bcf0d6135e1fb9804c1e81b36f60f832d552a76f407606ebebb9abd\": container with ID starting with 5150017d9bcf0d6135e1fb9804c1e81b36f60f832d552a76f407606ebebb9abd not found: ID does not exist"
Oct 31 23:14:25.084915 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:14:25.076653    4301 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10903-578db2\" is forbidden: User \"system:node:ip-10-0-135-163.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-163.us-east-2.compute.internal' and this object" podUID="a587bcee-0f91-41f9-beaa-b2de850c9119" pod="opendatahub/test-curl-10903-578db2"
Oct 31 23:14:25.084915 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:25.077901    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"]
Oct 31 23:14:25.229442 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:25.224589    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:14:25.229442 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:25.224694    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:14:25.229442 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:25.224737    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:14:25.507405 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:25.504655    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a587bcee-0f91-41f9-beaa-b2de850c9119" path="/var/lib/kubelet/pods/a587bcee-0f91-41f9-beaa-b2de850c9119/volumes"
Oct 31 23:14:25.836733 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:25.836687    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76" event={"ID":"3f037628-d51c-48e0-be4e-5d9c99a10332","Type":"ContainerStarted","Data":"be200d967c5ee63551ff33339c18d452825ab9f9a34a5f6752689f4b9bc81cd0"}
Oct 31 23:14:25.837211 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:25.836782    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76" event={"ID":"3f037628-d51c-48e0-be4e-5d9c99a10332","Type":"ContainerStarted","Data":"042b69f3e079a7911786bfedaa0f2581afe8fbaad6e018fb6d3ff968bc3c8069"}
Oct 31 23:14:25.839934 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:25.839902    4301 generic.go:356] "Generic (PLEG): container finished" podID="913b2d55-ba28-40c3-b19f-de5ee21ffe84" containerID="634ed05b296dee7ff8dd57a00e2ef566ca235bd70c9861e47891c9208fac6f82" exitCode=137
Oct 31 23:14:25.862409 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:25.862312    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76" podStartSLOduration=1.862292515 podStartE2EDuration="1.862292515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:14:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:14:25.858117829 +0000 UTC m=+1484.605307333" watchObservedRunningTime="2026-10-31 23:14:25.862292515 +0000 UTC m=+1484.609482021"
Oct 31 23:14:25.917653 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:25.917626    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10903-064925"
Oct 31 23:14:26.090126 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:26.090035    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/913b2d55-ba28-40c3-b19f-de5ee21ffe84-kube-api-access-8fwm5\" (UniqueName: \"kubernetes.io/projected/913b2d55-ba28-40c3-b19f-de5ee21ffe84-kube-api-access-8fwm5\") pod \"913b2d55-ba28-40c3-b19f-de5ee21ffe84\" (UID: \"913b2d55-ba28-40c3-b19f-de5ee21ffe84\") "
Oct 31 23:14:26.099037 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:26.098983    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/913b2d55-ba28-40c3-b19f-de5ee21ffe84-kube-api-access-8fwm5" pod "913b2d55-ba28-40c3-b19f-de5ee21ffe84" (UID: "913b2d55-ba28-40c3-b19f-de5ee21ffe84"). InnerVolumeSpecName "kube-api-access-8fwm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:14:26.209493 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:26.206349    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fwm5\" (UniqueName: \"kubernetes.io/projected/913b2d55-ba28-40c3-b19f-de5ee21ffe84-kube-api-access-8fwm5\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:14:26.462514 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:26.462475    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:14:26.468285 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:26.468259    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:14:26.847474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:26.847391    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10903-064925"
Oct 31 23:14:26.847474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:26.847425    4301 scope.go:122] "RemoveContainer" containerID="634ed05b296dee7ff8dd57a00e2ef566ca235bd70c9861e47891c9208fac6f82"
Oct 31 23:14:26.847973 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:26.847759    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:14:26.849159 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:26.849134    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:14:26.872329 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:14:26.872286    4301 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10903-064925\" is forbidden: User \"system:node:ip-10-0-135-163.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-163.us-east-2.compute.internal' and this object" podUID="913b2d55-ba28-40c3-b19f-de5ee21ffe84" pod="opendatahub/test-curl-10903-064925"
Oct 31 23:14:27.407331 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:27.407290    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="913b2d55-ba28-40c3-b19f-de5ee21ffe84" path="/var/lib/kubelet/pods/913b2d55-ba28-40c3-b19f-de5ee21ffe84/volumes"
Oct 31 23:14:28.306742 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:28.306702    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ed6117ae-69dd446956-qls9z"]
Oct 31 23:14:28.313281 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:28.313255    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ed6117ae-69dd446956-qls9z"
Oct 31 23:14:28.319327 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:28.319292    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ed6117ae-69dd446956-qls9z"]
Oct 31 23:14:28.385747 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:28.385712    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-ed6117ae\""
Oct 31 23:14:28.514379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:28.404722    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-ed6117ae-dockercfg-mfksb\""
Oct 31 23:14:28.514379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:28.405712    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w54ff\" (UniqueName: \"kubernetes.io/projected/4327b4bd-bb08-42da-933e-3e332b2eb85a-kube-api-access-w54ff\") pod \"payload-processing-e2e-migrate-ed6117ae-69dd446956-qls9z\" (UID: \"4327b4bd-bb08-42da-933e-3e332b2eb85a\") " pod="openshift-ingress/payload-processing-e2e-migrate-ed6117ae-69dd446956-qls9z"
Oct 31 23:14:28.514379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:28.451699    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4327b4bd-bb08-42da-933e-3e332b2eb85a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ed6117ae-69dd446956-qls9z\" (UID: \"4327b4bd-bb08-42da-933e-3e332b2eb85a\") " pod="openshift-ingress/payload-processing-e2e-migrate-ed6117ae-69dd446956-qls9z"
Oct 31 23:14:28.591489 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:28.555304    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4327b4bd-bb08-42da-933e-3e332b2eb85a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ed6117ae-69dd446956-qls9z\" (UID: \"4327b4bd-bb08-42da-933e-3e332b2eb85a\") " pod="openshift-ingress/payload-processing-e2e-migrate-ed6117ae-69dd446956-qls9z"
Oct 31 23:14:28.591489 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:28.555717    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w54ff\" (UniqueName: \"kubernetes.io/projected/4327b4bd-bb08-42da-933e-3e332b2eb85a-kube-api-access-w54ff\") pod \"payload-processing-e2e-migrate-ed6117ae-69dd446956-qls9z\" (UID: \"4327b4bd-bb08-42da-933e-3e332b2eb85a\") " pod="openshift-ingress/payload-processing-e2e-migrate-ed6117ae-69dd446956-qls9z"
Oct 31 23:14:28.591489 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:28.557009    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4327b4bd-bb08-42da-933e-3e332b2eb85a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ed6117ae-69dd446956-qls9z\" (UID: \"4327b4bd-bb08-42da-933e-3e332b2eb85a\") " pod="openshift-ingress/payload-processing-e2e-migrate-ed6117ae-69dd446956-qls9z"
Oct 31 23:14:28.688537 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:28.688347    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w54ff\" (UniqueName: \"kubernetes.io/projected/4327b4bd-bb08-42da-933e-3e332b2eb85a-kube-api-access-w54ff\") pod \"payload-processing-e2e-migrate-ed6117ae-69dd446956-qls9z\" (UID: \"4327b4bd-bb08-42da-933e-3e332b2eb85a\") " pod="openshift-ingress/payload-processing-e2e-migrate-ed6117ae-69dd446956-qls9z"
Oct 31 23:14:28.957849 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:28.957816    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ed6117ae-69dd446956-qls9z"
Oct 31 23:14:29.458801 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:29.452321    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ed6117ae-69dd446956-qls9z"]
Oct 31 23:14:30.031092 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:30.030996    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ed6117ae-69dd446956-qls9z" event={"ID":"4327b4bd-bb08-42da-933e-3e332b2eb85a","Type":"ContainerStarted","Data":"9540e55b69d2b2f2d550ca6b0e47219bae69c5aff7b6da5d0731a1f0b35ec3e0"}
Oct 31 23:14:31.151377 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:31.136883    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ed6117ae-69dd446956-qls9z" event={"ID":"4327b4bd-bb08-42da-933e-3e332b2eb85a","Type":"ContainerStarted","Data":"85f0f53d3d8bd6d0505bdf4e20e55545ea0171a1439f2a874d9c1039fc2ed5d4"}
Oct 31 23:14:31.151377 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:31.137176    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-ed6117ae-69dd446956-qls9z"
Oct 31 23:14:31.568747 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:31.568648    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:31.941574 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:31.941521    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:32.363620 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:32.363530    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 23:14:32.364569 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:32.364121    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 31 23:14:32.580393 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:32.573480    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:32.580393 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:32.574818    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 23:14:32.580393 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:32.575176    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:14:32.605077 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:32.600711    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-ed6117ae-69dd446956-qls9z" podStartSLOduration=4.600690473 podStartE2EDuration="4.600690473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:14:31.184699604 +0000 UTC m=+1489.931889111" watchObservedRunningTime="2026-10-31 23:14:32.600690473 +0000 UTC m=+1491.347879979"
Oct 31 23:14:32.671436 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:32.629499    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:32.846648 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:32.846601    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:32.870382 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:14:32.868618    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:14:33.007709 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:33.007612    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:33.029384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:33.011759    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 23:14:33.029384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:33.013002    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:14:33.382593 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:33.382536    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 31 23:14:33.397387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:33.392147    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:14:33.428527 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:33.428465    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:33.450475 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:33.446992    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:33.541386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:33.537692    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:33.552947 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:33.552650    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": context deadline exceeded"
Oct 31 23:14:33.587608 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:33.587567    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:33.592093 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:33.591105    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:33.617612 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:33.617578    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:14:33.617909 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:33.617893    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:14:33.644463 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:33.632436    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 23:14:33.701385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:33.690477    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:14:33.878778 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:33.878744    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:33.879822 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:33.878606    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:33.880186 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:33.880171    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:14:34.048459 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:34.044572    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 23:14:34.066417 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:34.052905    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:34.067000 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:34.066971    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:14:34.385726 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:34.385678    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:34.442382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:34.438456    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:34.442382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:34.438900    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:14:34.447617 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:34.447439    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 23:14:34.578521 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:34.578477    4301 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 23:14:34.578795 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:34.578765    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5" podUID="7a96c54a-c66d-466f-b2d4-84a208c3c664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:34.579077 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:34.579048    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 31 23:14:34.582061 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:34.582033    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:14:34.594472 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:34.594433    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 23:14:34.627269 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:34.607419    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:14:34.627269 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:34.608797    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:34.648921 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:34.648822    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:14:34.892388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:34.881192    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:34.959439 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:34.953481    4301 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 23:14:34.959439 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:34.953543    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm" podUID="d361d757-5e4e-43a4-acb7-9ef5f24113af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:35.185383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:35.181392    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:35.307590 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:35.237945    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 23:14:35.427517 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:35.427466    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:35.517395 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:35.455993    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:35.517395 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:35.456770    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:35.581485 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:35.580278    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:35.627694 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:35.627649    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:35.894977 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:35.892943    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:36.014126 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:36.014092    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:14:36.388465 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:36.321176    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:36.388465 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:36.322105    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 23:14:36.470615 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:36.470568    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:36.485435 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:36.473060    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:36.616843 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:36.616792    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:36.628452 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:36.627981    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:36.902382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:36.901478    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:36.926552 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:36.926496    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 23:14:36.928080 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:36.928039    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:14:36.992828 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:36.991660    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:14:37.149391 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:37.122647    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:14:37.203015 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:37.202918    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:14:37.273649 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:37.268484    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:14:37.273649 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:37.272067    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:14:37.344435 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:37.333320    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:14:37.367118 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:37.347636    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:14:37.525143 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:37.525036    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ac5ff20f-6fb5d9b-f9qlx"]
Oct 31 23:14:37.529606 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:37.529582    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ac5ff20f-6fb5d9b-f9qlx"
Oct 31 23:14:37.535739 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:37.535708    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ac5ff20f\""
Oct 31 23:14:37.541249 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:37.541206    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ac5ff20f-6fb5d9b-f9qlx"]
Oct 31 23:14:37.557785 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:37.557756    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ac5ff20f-6567b4b69f-sgrvt"]
Oct 31 23:14:37.562405 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:37.562382    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac5ff20f-6567b4b69f-sgrvt"
Oct 31 23:14:37.569169 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:37.569118    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ac5ff20f-dockercfg-vqhkf\""
Oct 31 23:14:37.572282 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:37.572254    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ac5ff20f-6567b4b69f-sgrvt"]
Oct 31 23:14:37.685384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:37.684447    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kcv5\" (UniqueName: \"kubernetes.io/projected/6eb38f55-705c-4b68-a700-9caa0396fc55-kube-api-access-9kcv5\") pod \"payload-pre-processing-e2e-ait-ac5ff20f-6567b4b69f-sgrvt\" (UID: \"6eb38f55-705c-4b68-a700-9caa0396fc55\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ac5ff20f-6567b4b69f-sgrvt"
Oct 31 23:14:37.685384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:37.684628    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eb38f55-705c-4b68-a700-9caa0396fc55-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ac5ff20f-6567b4b69f-sgrvt\" (UID: \"6eb38f55-705c-4b68-a700-9caa0396fc55\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ac5ff20f-6567b4b69f-sgrvt"
Oct 31 23:14:37.685384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:37.684709    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb917671-f386-4be1-99ca-8d7917d484cf-plugins-config-volume\") pod \"payload-processing-e2e-ait-ac5ff20f-6fb5d9b-f9qlx\" (UID: \"cb917671-f386-4be1-99ca-8d7917d484cf\") " pod="openshift-ingress/payload-processing-e2e-ait-ac5ff20f-6fb5d9b-f9qlx"
Oct 31 23:14:37.685384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:37.684743    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2m9n\" (UniqueName: \"kubernetes.io/projected/cb917671-f386-4be1-99ca-8d7917d484cf-kube-api-access-w2m9n\") pod \"payload-processing-e2e-ait-ac5ff20f-6fb5d9b-f9qlx\" (UID: \"cb917671-f386-4be1-99ca-8d7917d484cf\") " pod="openshift-ingress/payload-processing-e2e-ait-ac5ff20f-6fb5d9b-f9qlx"
Oct 31 23:14:37.787428 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:37.787304    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eb38f55-705c-4b68-a700-9caa0396fc55-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ac5ff20f-6567b4b69f-sgrvt\" (UID: \"6eb38f55-705c-4b68-a700-9caa0396fc55\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ac5ff20f-6567b4b69f-sgrvt"
Oct 31 23:14:37.787731 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:37.787710    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb917671-f386-4be1-99ca-8d7917d484cf-plugins-config-volume\") pod \"payload-processing-e2e-ait-ac5ff20f-6fb5d9b-f9qlx\" (UID: \"cb917671-f386-4be1-99ca-8d7917d484cf\") " pod="openshift-ingress/payload-processing-e2e-ait-ac5ff20f-6fb5d9b-f9qlx"
Oct 31 23:14:37.787855 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:37.787841    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2m9n\" (UniqueName: \"kubernetes.io/projected/cb917671-f386-4be1-99ca-8d7917d484cf-kube-api-access-w2m9n\") pod \"payload-processing-e2e-ait-ac5ff20f-6fb5d9b-f9qlx\" (UID: \"cb917671-f386-4be1-99ca-8d7917d484cf\") " pod="openshift-ingress/payload-processing-e2e-ait-ac5ff20f-6fb5d9b-f9qlx"
Oct 31 23:14:37.788240 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:37.788219    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kcv5\" (UniqueName: \"kubernetes.io/projected/6eb38f55-705c-4b68-a700-9caa0396fc55-kube-api-access-9kcv5\") pod \"payload-pre-processing-e2e-ait-ac5ff20f-6567b4b69f-sgrvt\" (UID: \"6eb38f55-705c-4b68-a700-9caa0396fc55\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ac5ff20f-6567b4b69f-sgrvt"
Oct 31 23:14:37.800394 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:37.789842    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eb38f55-705c-4b68-a700-9caa0396fc55-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ac5ff20f-6567b4b69f-sgrvt\" (UID: \"6eb38f55-705c-4b68-a700-9caa0396fc55\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ac5ff20f-6567b4b69f-sgrvt"
Oct 31 23:14:37.802458 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:37.801535    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb917671-f386-4be1-99ca-8d7917d484cf-plugins-config-volume\") pod \"payload-processing-e2e-ait-ac5ff20f-6fb5d9b-f9qlx\" (UID: \"cb917671-f386-4be1-99ca-8d7917d484cf\") " pod="openshift-ingress/payload-processing-e2e-ait-ac5ff20f-6fb5d9b-f9qlx"
Oct 31 23:14:37.809958 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:37.809927    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kcv5\" (UniqueName: \"kubernetes.io/projected/6eb38f55-705c-4b68-a700-9caa0396fc55-kube-api-access-9kcv5\") pod \"payload-pre-processing-e2e-ait-ac5ff20f-6567b4b69f-sgrvt\" (UID: \"6eb38f55-705c-4b68-a700-9caa0396fc55\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ac5ff20f-6567b4b69f-sgrvt"
Oct 31 23:14:37.839112 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:37.839075    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2m9n\" (UniqueName: \"kubernetes.io/projected/cb917671-f386-4be1-99ca-8d7917d484cf-kube-api-access-w2m9n\") pod \"payload-processing-e2e-ait-ac5ff20f-6fb5d9b-f9qlx\" (UID: \"cb917671-f386-4be1-99ca-8d7917d484cf\") " pod="openshift-ingress/payload-processing-e2e-ait-ac5ff20f-6fb5d9b-f9qlx"
Oct 31 23:14:37.853331 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:37.853297    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ac5ff20f-6fb5d9b-f9qlx"
Oct 31 23:14:37.886188 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:37.886157    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac5ff20f-6567b4b69f-sgrvt"
Oct 31 23:14:38.271025 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:38.271002    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ac5ff20f-6fb5d9b-f9qlx"]
Oct 31 23:14:38.272742 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:14:38.272714    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb917671_f386_4be1_99ca_8d7917d484cf.slice/crio-77801ee97e40580c5184a9e3c1ca726319001de0cbf8c7b9f3d72e1e83f9799b WatchSource:0}: Error finding container 77801ee97e40580c5184a9e3c1ca726319001de0cbf8c7b9f3d72e1e83f9799b: Status 404 returned error can't find the container with id 77801ee97e40580c5184a9e3c1ca726319001de0cbf8c7b9f3d72e1e83f9799b
Oct 31 23:14:38.281145 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:38.281119    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ac5ff20f-6567b4b69f-sgrvt"]
Oct 31 23:14:38.284402 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:14:38.284338    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb38f55_705c_4b68_a700_9caa0396fc55.slice/crio-1b171859504eb3a925e0203d67cdb5669aa85e364096391dccefdf76185f45f7 WatchSource:0}: Error finding container 1b171859504eb3a925e0203d67cdb5669aa85e364096391dccefdf76185f45f7: Status 404 returned error can't find the container with id 1b171859504eb3a925e0203d67cdb5669aa85e364096391dccefdf76185f45f7
Oct 31 23:14:38.967538 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:38.967502    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac5ff20f-6567b4b69f-sgrvt" event={"ID":"6eb38f55-705c-4b68-a700-9caa0396fc55","Type":"ContainerStarted","Data":"3dbf47a170c027a2c94fdb4176a6706d4a5a1f3c25c97a784936357026ed438c"}
Oct 31 23:14:38.968008 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:38.967564    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac5ff20f-6567b4b69f-sgrvt"
Oct 31 23:14:38.968008 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:38.967584    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac5ff20f-6567b4b69f-sgrvt" event={"ID":"6eb38f55-705c-4b68-a700-9caa0396fc55","Type":"ContainerStarted","Data":"1b171859504eb3a925e0203d67cdb5669aa85e364096391dccefdf76185f45f7"}
Oct 31 23:14:38.969167 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:38.969138    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ac5ff20f-6fb5d9b-f9qlx" event={"ID":"cb917671-f386-4be1-99ca-8d7917d484cf","Type":"ContainerStarted","Data":"59d1df9cd0efae0f40bb426b078a5853051fa62cf5834c7401387bcd524543ad"}
Oct 31 23:14:38.969268 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:38.969184    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ac5ff20f-6fb5d9b-f9qlx" event={"ID":"cb917671-f386-4be1-99ca-8d7917d484cf","Type":"ContainerStarted","Data":"77801ee97e40580c5184a9e3c1ca726319001de0cbf8c7b9f3d72e1e83f9799b"}
Oct 31 23:14:38.969268 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:38.969204    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ac5ff20f-6fb5d9b-f9qlx"
Oct 31 23:14:38.987493 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:38.987434    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac5ff20f-6567b4b69f-sgrvt" podStartSLOduration=1.987421906 podStartE2EDuration="1.987421906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:14:38.983869377 +0000 UTC m=+1497.731058882" watchObservedRunningTime="2026-10-31 23:14:38.987421906 +0000 UTC m=+1497.734611410"
Oct 31 23:14:39.004194 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:39.004152    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ac5ff20f-6fb5d9b-f9qlx" podStartSLOduration=2.00413681 podStartE2EDuration="2.00413681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:14:39.000594478 +0000 UTC m=+1497.747784015" watchObservedRunningTime="2026-10-31 23:14:39.00413681 +0000 UTC m=+1497.751326370"
Oct 31 23:14:40.603179 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:40.603141    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4"]
Oct 31 23:14:40.608556 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:40.608528    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4"
Oct 31 23:14:40.627497 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:40.625813    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4"]
Oct 31 23:14:40.627497 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:40.626323    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-ed6117ae-serving-cert\""
Oct 31 23:14:40.765383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:40.763009    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72a95a93-f42b-4829-aaf5-b388a7482e8e-maas-api-tls\") pod \"maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4\" (UID: \"72a95a93-f42b-4829-aaf5-b388a7482e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4"
Oct 31 23:14:40.765383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:40.763273    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72a95a93-f42b-4829-aaf5-b388a7482e8e-openshift-service-ca\") pod \"maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4\" (UID: \"72a95a93-f42b-4829-aaf5-b388a7482e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4"
Oct 31 23:14:40.765383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:40.763378    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72a95a93-f42b-4829-aaf5-b388a7482e8e-metrics-certs\") pod \"maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4\" (UID: \"72a95a93-f42b-4829-aaf5-b388a7482e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4"
Oct 31 23:14:40.765383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:40.763407    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlhqz\" (UniqueName: \"kubernetes.io/projected/72a95a93-f42b-4829-aaf5-b388a7482e8e-kube-api-access-zlhqz\") pod \"maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4\" (UID: \"72a95a93-f42b-4829-aaf5-b388a7482e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4"
Oct 31 23:14:40.872184 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:40.870585    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72a95a93-f42b-4829-aaf5-b388a7482e8e-maas-api-tls\") pod \"maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4\" (UID: \"72a95a93-f42b-4829-aaf5-b388a7482e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4"
Oct 31 23:14:40.872184 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:40.870827    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72a95a93-f42b-4829-aaf5-b388a7482e8e-openshift-service-ca\") pod \"maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4\" (UID: \"72a95a93-f42b-4829-aaf5-b388a7482e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4"
Oct 31 23:14:40.872184 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:40.870913    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72a95a93-f42b-4829-aaf5-b388a7482e8e-metrics-certs\") pod \"maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4\" (UID: \"72a95a93-f42b-4829-aaf5-b388a7482e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4"
Oct 31 23:14:40.872184 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:40.870942    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlhqz\" (UniqueName: \"kubernetes.io/projected/72a95a93-f42b-4829-aaf5-b388a7482e8e-kube-api-access-zlhqz\") pod \"maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4\" (UID: \"72a95a93-f42b-4829-aaf5-b388a7482e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4"
Oct 31 23:14:40.872184 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:14:40.871164    4301 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-ed6117ae-serving-cert: secret "maas-api-e2e-migrate-ed6117ae-serving-cert" not found
Oct 31 23:14:40.872184 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:14:40.871237    4301 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72a95a93-f42b-4829-aaf5-b388a7482e8e-maas-api-tls podName:72a95a93-f42b-4829-aaf5-b388a7482e8e nodeName:}" failed. No retries permitted until 2026-10-31 23:14:41.371211205 +0000 UTC m=+1500.118400710 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/72a95a93-f42b-4829-aaf5-b388a7482e8e-maas-api-tls") pod "maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4" (UID: "72a95a93-f42b-4829-aaf5-b388a7482e8e") : secret "maas-api-e2e-migrate-ed6117ae-serving-cert" not found
Oct 31 23:14:40.872664 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:40.872243    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72a95a93-f42b-4829-aaf5-b388a7482e8e-openshift-service-ca\") pod \"maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4\" (UID: \"72a95a93-f42b-4829-aaf5-b388a7482e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4"
Oct 31 23:14:40.876401 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:40.875376    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72a95a93-f42b-4829-aaf5-b388a7482e8e-metrics-certs\") pod \"maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4\" (UID: \"72a95a93-f42b-4829-aaf5-b388a7482e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4"
Oct 31 23:14:40.884250 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:40.884226    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlhqz\" (UniqueName: \"kubernetes.io/projected/72a95a93-f42b-4829-aaf5-b388a7482e8e-kube-api-access-zlhqz\") pod \"maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4\" (UID: \"72a95a93-f42b-4829-aaf5-b388a7482e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4"
Oct 31 23:14:41.383997 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:41.383964    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72a95a93-f42b-4829-aaf5-b388a7482e8e-maas-api-tls\") pod \"maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4\" (UID: \"72a95a93-f42b-4829-aaf5-b388a7482e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4"
Oct 31 23:14:41.386983 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:41.386956    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72a95a93-f42b-4829-aaf5-b388a7482e8e-maas-api-tls\") pod \"maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4\" (UID: \"72a95a93-f42b-4829-aaf5-b388a7482e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4"
Oct 31 23:14:41.567384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:41.558219    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4"
Oct 31 23:14:41.660461 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:41.660430    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s9nl_1f0042ee-6258-4aa8-af48-7568f2f2e8a1/ovn-acl-logging/0.log"
Oct 31 23:14:41.669142 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:41.669116    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s9nl_1f0042ee-6258-4aa8-af48-7568f2f2e8a1/ovn-acl-logging/0.log"
Oct 31 23:14:41.915107 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:41.915079    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4"]
Oct 31 23:14:41.917041 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:14:41.917011    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72a95a93_f42b_4829_aaf5_b388a7482e8e.slice/crio-b3d55fc6c877a1a6427fd92d4aa7b9d3fac3325d31db29eacb3322752c794584 WatchSource:0}: Error finding container b3d55fc6c877a1a6427fd92d4aa7b9d3fac3325d31db29eacb3322752c794584: Status 404 returned error can't find the container with id b3d55fc6c877a1a6427fd92d4aa7b9d3fac3325d31db29eacb3322752c794584
Oct 31 23:14:42.002323 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:42.002291    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4" event={"ID":"72a95a93-f42b-4829-aaf5-b388a7482e8e","Type":"ContainerStarted","Data":"b3d55fc6c877a1a6427fd92d4aa7b9d3fac3325d31db29eacb3322752c794584"}
Oct 31 23:14:42.205737 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:42.205648    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-ed6117ae-69dd446956-qls9z"
Oct 31 23:14:43.039159 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:43.039093    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4" event={"ID":"72a95a93-f42b-4829-aaf5-b388a7482e8e","Type":"ContainerStarted","Data":"51fc005f0ef0c2d14e8e185999bef18f506c7d850e4a71cd6dea0e7247b47e54"}
Oct 31 23:14:43.041896 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:43.041873    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4"
Oct 31 23:14:43.065497 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:43.065440    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4" podStartSLOduration=2.677348692 podStartE2EDuration="3.065421445s" totalImagesPullingTime="388.072753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:14:43.065080925 +0000 UTC m=+1501.812270434" watchObservedRunningTime="2026-10-31 23:14:43.065421445 +0000 UTC m=+1501.812610946"
Oct 31 23:14:49.991784 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:49.991753    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac5ff20f-6567b4b69f-sgrvt"
Oct 31 23:14:49.992280 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:49.991818    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ac5ff20f-6fb5d9b-f9qlx"
Oct 31 23:14:50.095975 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:50.095947    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4"
Oct 31 23:14:57.393069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:57.393033    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b"]
Oct 31 23:14:57.400092 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:57.400066    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b"
Oct 31 23:14:57.417929 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:57.417900    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b"]
Oct 31 23:14:57.562380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:57.549409    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b331f886-ca34-4c4b-a31c-d2e45aac6af3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pvl9b\" (UID: \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b"
Oct 31 23:14:57.562380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:57.549503    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b331f886-ca34-4c4b-a31c-d2e45aac6af3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pvl9b\" (UID: \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b"
Oct 31 23:14:57.562380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:57.549557    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b331f886-ca34-4c4b-a31c-d2e45aac6af3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pvl9b\" (UID: \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b"
Oct 31 23:14:57.562380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:57.549613    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvfzz\" (UniqueName: \"kubernetes.io/projected/b331f886-ca34-4c4b-a31c-d2e45aac6af3-kube-api-access-fvfzz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pvl9b\" (UID: \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b"
Oct 31 23:14:57.562380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:57.549765    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b331f886-ca34-4c4b-a31c-d2e45aac6af3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pvl9b\" (UID: \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b"
Oct 31 23:14:57.562380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:57.550021    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b331f886-ca34-4c4b-a31c-d2e45aac6af3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pvl9b\" (UID: \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b"
Oct 31 23:14:57.562380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:57.550071    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b331f886-ca34-4c4b-a31c-d2e45aac6af3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pvl9b\" (UID: \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b"
Oct 31 23:14:57.653472 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:57.652313    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b331f886-ca34-4c4b-a31c-d2e45aac6af3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pvl9b\" (UID: \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b"
Oct 31 23:14:57.653472 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:57.652408    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b331f886-ca34-4c4b-a31c-d2e45aac6af3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pvl9b\" (UID: \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b"
Oct 31 23:14:57.653472 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:57.652492    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b331f886-ca34-4c4b-a31c-d2e45aac6af3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pvl9b\" (UID: \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b"
Oct 31 23:14:57.653472 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:57.652554    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b331f886-ca34-4c4b-a31c-d2e45aac6af3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pvl9b\" (UID: \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b"
Oct 31 23:14:57.653472 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:57.652596    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b331f886-ca34-4c4b-a31c-d2e45aac6af3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pvl9b\" (UID: \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b"
Oct 31 23:14:57.653472 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:57.652648    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvfzz\" (UniqueName: \"kubernetes.io/projected/b331f886-ca34-4c4b-a31c-d2e45aac6af3-kube-api-access-fvfzz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pvl9b\" (UID: \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b"
Oct 31 23:14:57.653472 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:57.652781    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b331f886-ca34-4c4b-a31c-d2e45aac6af3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pvl9b\" (UID: \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b"
Oct 31 23:14:57.661209 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:57.661179    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b331f886-ca34-4c4b-a31c-d2e45aac6af3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pvl9b\" (UID: \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b"
Oct 31 23:14:57.662199 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:57.662173    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b331f886-ca34-4c4b-a31c-d2e45aac6af3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pvl9b\" (UID: \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b"
Oct 31 23:14:57.662858 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:57.662834    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b331f886-ca34-4c4b-a31c-d2e45aac6af3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pvl9b\" (UID: \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b"
Oct 31 23:14:57.663246 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:57.663226    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b331f886-ca34-4c4b-a31c-d2e45aac6af3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pvl9b\" (UID: \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b"
Oct 31 23:14:57.665784 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:57.665762    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b331f886-ca34-4c4b-a31c-d2e45aac6af3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pvl9b\" (UID: \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b"
Oct 31 23:14:57.667443 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:57.667424    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvfzz\" (UniqueName: \"kubernetes.io/projected/b331f886-ca34-4c4b-a31c-d2e45aac6af3-kube-api-access-fvfzz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pvl9b\" (UID: \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b"
Oct 31 23:14:57.667693 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:57.667679    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b331f886-ca34-4c4b-a31c-d2e45aac6af3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pvl9b\" (UID: \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b"
Oct 31 23:14:57.717351 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:57.717328    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b"
Oct 31 23:14:58.006607 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:58.006529    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b"]
Oct 31 23:14:58.010428 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:14:58.010338    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb331f886_ca34_4c4b_a31c_d2e45aac6af3.slice/crio-0ee0dfafce6f405cc36f5d3ae3f3d1f22bb11c7eb9070b03900e58d01335014f WatchSource:0}: Error finding container 0ee0dfafce6f405cc36f5d3ae3f3d1f22bb11c7eb9070b03900e58d01335014f: Status 404 returned error can't find the container with id 0ee0dfafce6f405cc36f5d3ae3f3d1f22bb11c7eb9070b03900e58d01335014f
Oct 31 23:14:58.013225 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:58.013186    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:14:58.013309 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:58.013257    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:14:58.271883 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:58.271795    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b" event={"ID":"b331f886-ca34-4c4b-a31c-d2e45aac6af3","Type":"ContainerStarted","Data":"ac276961c29947a2357957adc6b3d71e34bece94120ed88885ba0df52407edc9"}
Oct 31 23:14:58.271883 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:58.271854    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b" event={"ID":"b331f886-ca34-4c4b-a31c-d2e45aac6af3","Type":"ContainerStarted","Data":"0ee0dfafce6f405cc36f5d3ae3f3d1f22bb11c7eb9070b03900e58d01335014f"}
Oct 31 23:14:58.272608 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:58.272584    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b"
Oct 31 23:14:58.295805 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:58.295757    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b" podStartSLOduration=1.295742666 podStartE2EDuration="1.295742666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:14:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:14:58.294230821 +0000 UTC m=+1517.041420325" watchObservedRunningTime="2026-10-31 23:14:58.295742666 +0000 UTC m=+1517.042932171"
Oct 31 23:14:59.280335 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:14:59.280305    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b"
Oct 31 23:15:00.251085 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.251046    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg"]
Oct 31 23:15:00.255990 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.255968    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg"
Oct 31 23:15:00.288551 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.288510    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg"]
Oct 31 23:15:00.306256 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.306230    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\""
Oct 31 23:15:00.312920 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.312886    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b2c6553b-29891475-h4rm9"]
Oct 31 23:15:00.318323 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.318208    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b2c6553b-29891475-h4rm9"
Oct 31 23:15:00.320285 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.320263    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ed6117ae-29891475-7vdd9"]
Oct 31 23:15:00.326516 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.326488    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ed6117ae-29891475-7vdd9"
Oct 31 23:15:00.361225 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.361192    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ed6117ae-29891475-7vdd9"]
Oct 31 23:15:00.374225 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.374187    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b2c6553b-29891475-h4rm9"]
Oct 31 23:15:00.374380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.374264    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8"]
Oct 31 23:15:00.379588 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.379568    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8"
Oct 31 23:15:00.397071 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.397041    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8"]
Oct 31 23:15:00.408767 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.408735    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/865bc4e8-2471-4a65-b11e-5ebc94b58fee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-b2c6553b-29891475-h4rm9\" (UID: \"865bc4e8-2471-4a65-b11e-5ebc94b58fee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b2c6553b-29891475-h4rm9"
Oct 31 23:15:00.408902 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.408879    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dng2l\" (UniqueName: \"kubernetes.io/projected/09828acd-1f45-4f0b-b2df-dce7f38c2e3d-kube-api-access-dng2l\") pod \"maas-api-key-cleanup-e2e-migrate-ed6117ae-29891475-7vdd9\" (UID: \"09828acd-1f45-4f0b-b2df-dce7f38c2e3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ed6117ae-29891475-7vdd9"
Oct 31 23:15:00.409033 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.409002    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcv9n\" (UniqueName: \"kubernetes.io/projected/865bc4e8-2471-4a65-b11e-5ebc94b58fee-kube-api-access-tcv9n\") pod \"maas-api-key-cleanup-e2e-ait-b2c6553b-29891475-h4rm9\" (UID: \"865bc4e8-2471-4a65-b11e-5ebc94b58fee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b2c6553b-29891475-h4rm9"
Oct 31 23:15:00.409166 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.409081    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-677kj\" (UniqueName: \"kubernetes.io/projected/9c5d8c65-bbe8-486d-b24c-8f6a188f52c3-kube-api-access-677kj\") pod \"maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg\" (UID: \"9c5d8c65-bbe8-486d-b24c-8f6a188f52c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg"
Oct 31 23:15:00.409374 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.409334    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09828acd-1f45-4f0b-b2df-dce7f38c2e3d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-ed6117ae-29891475-7vdd9\" (UID: \"09828acd-1f45-4f0b-b2df-dce7f38c2e3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ed6117ae-29891475-7vdd9"
Oct 31 23:15:00.409594 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.409575    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c5d8c65-bbe8-486d-b24c-8f6a188f52c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg\" (UID: \"9c5d8c65-bbe8-486d-b24c-8f6a188f52c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg"
Oct 31 23:15:00.510874 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.510787    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c5d8c65-bbe8-486d-b24c-8f6a188f52c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg\" (UID: \"9c5d8c65-bbe8-486d-b24c-8f6a188f52c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg"
Oct 31 23:15:00.510874 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.510859    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/865bc4e8-2471-4a65-b11e-5ebc94b58fee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-b2c6553b-29891475-h4rm9\" (UID: \"865bc4e8-2471-4a65-b11e-5ebc94b58fee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b2c6553b-29891475-h4rm9"
Oct 31 23:15:00.511104 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.510918    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dng2l\" (UniqueName: \"kubernetes.io/projected/09828acd-1f45-4f0b-b2df-dce7f38c2e3d-kube-api-access-dng2l\") pod \"maas-api-key-cleanup-e2e-migrate-ed6117ae-29891475-7vdd9\" (UID: \"09828acd-1f45-4f0b-b2df-dce7f38c2e3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ed6117ae-29891475-7vdd9"
Oct 31 23:15:00.511104 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.510973    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcv9n\" (UniqueName: \"kubernetes.io/projected/865bc4e8-2471-4a65-b11e-5ebc94b58fee-kube-api-access-tcv9n\") pod \"maas-api-key-cleanup-e2e-ait-b2c6553b-29891475-h4rm9\" (UID: \"865bc4e8-2471-4a65-b11e-5ebc94b58fee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b2c6553b-29891475-h4rm9"
Oct 31 23:15:00.511104 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.511007    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-677kj\" (UniqueName: \"kubernetes.io/projected/9c5d8c65-bbe8-486d-b24c-8f6a188f52c3-kube-api-access-677kj\") pod \"maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg\" (UID: \"9c5d8c65-bbe8-486d-b24c-8f6a188f52c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg"
Oct 31 23:15:00.511104 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.511036    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09828acd-1f45-4f0b-b2df-dce7f38c2e3d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-ed6117ae-29891475-7vdd9\" (UID: \"09828acd-1f45-4f0b-b2df-dce7f38c2e3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ed6117ae-29891475-7vdd9"
Oct 31 23:15:00.511104 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.511092    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4xjh\" (UniqueName: \"kubernetes.io/projected/ce874bef-ee28-46ca-aacc-bb04f21486ea-kube-api-access-x4xjh\") pod \"maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8\" (UID: \"ce874bef-ee28-46ca-aacc-bb04f21486ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8"
Oct 31 23:15:00.511441 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.511194    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce874bef-ee28-46ca-aacc-bb04f21486ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8\" (UID: \"ce874bef-ee28-46ca-aacc-bb04f21486ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8"
Oct 31 23:15:00.511630 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.511607    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c5d8c65-bbe8-486d-b24c-8f6a188f52c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg\" (UID: \"9c5d8c65-bbe8-486d-b24c-8f6a188f52c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg"
Oct 31 23:15:00.511698 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.511652    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/865bc4e8-2471-4a65-b11e-5ebc94b58fee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-b2c6553b-29891475-h4rm9\" (UID: \"865bc4e8-2471-4a65-b11e-5ebc94b58fee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b2c6553b-29891475-h4rm9"
Oct 31 23:15:00.511755 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.511707    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09828acd-1f45-4f0b-b2df-dce7f38c2e3d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-ed6117ae-29891475-7vdd9\" (UID: \"09828acd-1f45-4f0b-b2df-dce7f38c2e3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ed6117ae-29891475-7vdd9"
Oct 31 23:15:00.523238 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.523215    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dng2l\" (UniqueName: \"kubernetes.io/projected/09828acd-1f45-4f0b-b2df-dce7f38c2e3d-kube-api-access-dng2l\") pod \"maas-api-key-cleanup-e2e-migrate-ed6117ae-29891475-7vdd9\" (UID: \"09828acd-1f45-4f0b-b2df-dce7f38c2e3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ed6117ae-29891475-7vdd9"
Oct 31 23:15:00.523339 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.523223    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcv9n\" (UniqueName: \"kubernetes.io/projected/865bc4e8-2471-4a65-b11e-5ebc94b58fee-kube-api-access-tcv9n\") pod \"maas-api-key-cleanup-e2e-ait-b2c6553b-29891475-h4rm9\" (UID: \"865bc4e8-2471-4a65-b11e-5ebc94b58fee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b2c6553b-29891475-h4rm9"
Oct 31 23:15:00.523415 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.523395    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-677kj\" (UniqueName: \"kubernetes.io/projected/9c5d8c65-bbe8-486d-b24c-8f6a188f52c3-kube-api-access-677kj\") pod \"maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg\" (UID: \"9c5d8c65-bbe8-486d-b24c-8f6a188f52c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg"
Oct 31 23:15:00.573408 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.573353    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg"
Oct 31 23:15:00.612177 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.612151    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce874bef-ee28-46ca-aacc-bb04f21486ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8\" (UID: \"ce874bef-ee28-46ca-aacc-bb04f21486ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8"
Oct 31 23:15:00.612553 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.612514    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4xjh\" (UniqueName: \"kubernetes.io/projected/ce874bef-ee28-46ca-aacc-bb04f21486ea-kube-api-access-x4xjh\") pod \"maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8\" (UID: \"ce874bef-ee28-46ca-aacc-bb04f21486ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8"
Oct 31 23:15:00.612988 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.612967    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce874bef-ee28-46ca-aacc-bb04f21486ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8\" (UID: \"ce874bef-ee28-46ca-aacc-bb04f21486ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8"
Oct 31 23:15:00.624019 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.623996    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4xjh\" (UniqueName: \"kubernetes.io/projected/ce874bef-ee28-46ca-aacc-bb04f21486ea-kube-api-access-x4xjh\") pod \"maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8\" (UID: \"ce874bef-ee28-46ca-aacc-bb04f21486ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8"
Oct 31 23:15:00.648547 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.648512    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b2c6553b-29891475-h4rm9"
Oct 31 23:15:00.657851 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.657679    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ed6117ae-29891475-7vdd9"
Oct 31 23:15:00.706035 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.705794    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8"
Oct 31 23:15:00.862875 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.862632    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg"]
Oct 31 23:15:00.863004 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:15:00.862924    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c5d8c65_bbe8_486d_b24c_8f6a188f52c3.slice/crio-45127ad2de8f00bdd7c8e2fa8dba43e38426a3b9de696fd2ff8657108f1a0f5f WatchSource:0}: Error finding container 45127ad2de8f00bdd7c8e2fa8dba43e38426a3b9de696fd2ff8657108f1a0f5f: Status 404 returned error can't find the container with id 45127ad2de8f00bdd7c8e2fa8dba43e38426a3b9de696fd2ff8657108f1a0f5f
Oct 31 23:15:00.949281 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.949249    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b2c6553b-29891475-h4rm9"]
Oct 31 23:15:00.949826 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:15:00.949801    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod865bc4e8_2471_4a65_b11e_5ebc94b58fee.slice/crio-7c53c91b46e42899406abe6c050255079b2613c39e507aa2bd56539c175f1b6b WatchSource:0}: Error finding container 7c53c91b46e42899406abe6c050255079b2613c39e507aa2bd56539c175f1b6b: Status 404 returned error can't find the container with id 7c53c91b46e42899406abe6c050255079b2613c39e507aa2bd56539c175f1b6b
Oct 31 23:15:00.968789 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:00.968752    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ed6117ae-29891475-7vdd9"]
Oct 31 23:15:00.969616 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:15:00.969565    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09828acd_1f45_4f0b_b2df_dce7f38c2e3d.slice/crio-2f307cad85e080f684e6d54cdb70034339491d7b68e78df93590c25b28d53d2d WatchSource:0}: Error finding container 2f307cad85e080f684e6d54cdb70034339491d7b68e78df93590c25b28d53d2d: Status 404 returned error can't find the container with id 2f307cad85e080f684e6d54cdb70034339491d7b68e78df93590c25b28d53d2d
Oct 31 23:15:01.091890 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:01.072064    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8"]
Oct 31 23:15:01.294174 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:01.294134    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b2c6553b-29891475-h4rm9" event={"ID":"865bc4e8-2471-4a65-b11e-5ebc94b58fee","Type":"ContainerStarted","Data":"7c53c91b46e42899406abe6c050255079b2613c39e507aa2bd56539c175f1b6b"}
Oct 31 23:15:01.295463 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:01.295436    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8" event={"ID":"ce874bef-ee28-46ca-aacc-bb04f21486ea","Type":"ContainerStarted","Data":"9d156cc3183005001b3ae0f8cc80120126f1c09e2e0c205474c20a2f035a4368"}
Oct 31 23:15:01.296857 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:01.296834    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg" event={"ID":"9c5d8c65-bbe8-486d-b24c-8f6a188f52c3","Type":"ContainerStarted","Data":"9c9a5eb3b0d2f7c57735eff2db1f3b9fb860a60c0698d362c020999b389655cb"}
Oct 31 23:15:01.296972 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:01.296878    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg" event={"ID":"9c5d8c65-bbe8-486d-b24c-8f6a188f52c3","Type":"ContainerStarted","Data":"45127ad2de8f00bdd7c8e2fa8dba43e38426a3b9de696fd2ff8657108f1a0f5f"}
Oct 31 23:15:01.297990 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:01.297970    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ed6117ae-29891475-7vdd9" event={"ID":"09828acd-1f45-4f0b-b2df-dce7f38c2e3d","Type":"ContainerStarted","Data":"2f307cad85e080f684e6d54cdb70034339491d7b68e78df93590c25b28d53d2d"}
Oct 31 23:15:01.318869 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:01.318787    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg" podStartSLOduration=1.318770839 podStartE2EDuration="1.318770839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:15:01.31410108 +0000 UTC m=+1520.061290584" watchObservedRunningTime="2026-10-31 23:15:01.318770839 +0000 UTC m=+1520.065960345"
Oct 31 23:15:02.305732 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:02.305648    4301 generic.go:356] "Generic (PLEG): container finished" podID="09828acd-1f45-4f0b-b2df-dce7f38c2e3d" containerID="5a54f187f5985988803a7f7beccac6063dbbee4e6449ca71994383d62c031bb7" exitCode=60
Oct 31 23:15:02.306173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:02.305736    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ed6117ae-29891475-7vdd9" event={"ID":"09828acd-1f45-4f0b-b2df-dce7f38c2e3d","Type":"ContainerDied","Data":"5a54f187f5985988803a7f7beccac6063dbbee4e6449ca71994383d62c031bb7"}
Oct 31 23:15:02.306173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:02.306050    4301 scope.go:122] "RemoveContainer" containerID="5a54f187f5985988803a7f7beccac6063dbbee4e6449ca71994383d62c031bb7"
Oct 31 23:15:02.307270 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:02.307247    4301 generic.go:356] "Generic (PLEG): container finished" podID="865bc4e8-2471-4a65-b11e-5ebc94b58fee" containerID="f09f447a3eae0b234fdf487ca9ab1cca761f23ad90b6913fbad0b9b503d82e80" exitCode=60
Oct 31 23:15:02.307415 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:02.307389    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b2c6553b-29891475-h4rm9" event={"ID":"865bc4e8-2471-4a65-b11e-5ebc94b58fee","Type":"ContainerDied","Data":"f09f447a3eae0b234fdf487ca9ab1cca761f23ad90b6913fbad0b9b503d82e80"}
Oct 31 23:15:02.307613 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:02.307598    4301 scope.go:122] "RemoveContainer" containerID="f09f447a3eae0b234fdf487ca9ab1cca761f23ad90b6913fbad0b9b503d82e80"
Oct 31 23:15:02.316117 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:02.316090    4301 generic.go:356] "Generic (PLEG): container finished" podID="ce874bef-ee28-46ca-aacc-bb04f21486ea" containerID="2e469015c9cc62cff665b7eab2ed713b53d39d5a0c56ef5c21dd112316f122e0" exitCode=60
Oct 31 23:15:02.316228 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:02.316173    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8" event={"ID":"ce874bef-ee28-46ca-aacc-bb04f21486ea","Type":"ContainerDied","Data":"2e469015c9cc62cff665b7eab2ed713b53d39d5a0c56ef5c21dd112316f122e0"}
Oct 31 23:15:02.316465 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:02.316446    4301 scope.go:122] "RemoveContainer" containerID="2e469015c9cc62cff665b7eab2ed713b53d39d5a0c56ef5c21dd112316f122e0"
Oct 31 23:15:02.318662 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:02.318640    4301 generic.go:356] "Generic (PLEG): container finished" podID="9c5d8c65-bbe8-486d-b24c-8f6a188f52c3" containerID="9c9a5eb3b0d2f7c57735eff2db1f3b9fb860a60c0698d362c020999b389655cb" exitCode=60
Oct 31 23:15:02.318756 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:02.318698    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg" event={"ID":"9c5d8c65-bbe8-486d-b24c-8f6a188f52c3","Type":"ContainerDied","Data":"9c9a5eb3b0d2f7c57735eff2db1f3b9fb860a60c0698d362c020999b389655cb"}
Oct 31 23:15:02.319011 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:02.318996    4301 scope.go:122] "RemoveContainer" containerID="9c9a5eb3b0d2f7c57735eff2db1f3b9fb860a60c0698d362c020999b389655cb"
Oct 31 23:15:02.744427 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:15:02.744369    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:15:03.121394 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:03.121348    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ed6117ae-69dd446956-qls9z"]
Oct 31 23:15:03.327276 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:03.327191    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ed6117ae-29891475-7vdd9" event={"ID":"09828acd-1f45-4f0b-b2df-dce7f38c2e3d","Type":"ContainerStarted","Data":"45bd8808074a3e3436a56c41e6ed526f6562e992ac2b5037d398b3449322a9cd"}
Oct 31 23:15:03.329536 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:03.329345    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b2c6553b-29891475-h4rm9" event={"ID":"865bc4e8-2471-4a65-b11e-5ebc94b58fee","Type":"ContainerStarted","Data":"038f00ba7af0a0ee2169df172102f4fe513e8f4518cce01fccdbc49a188241a5"}
Oct 31 23:15:03.333228 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:03.333205    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8" event={"ID":"ce874bef-ee28-46ca-aacc-bb04f21486ea","Type":"ContainerStarted","Data":"d4a9bd6166532e79e924e4d9fdfa51015ab608f46cbf0bfbf8e7100478b9622b"}
Oct 31 23:15:03.335692 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:03.335664    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg" event={"ID":"9c5d8c65-bbe8-486d-b24c-8f6a188f52c3","Type":"ContainerStarted","Data":"8119859baebf4034b1a2ca8b9104c18133bf695404b8c78929c3eae310b97dbf"}
Oct 31 23:15:03.346680 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:03.346640    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ed6117ae-29891475-7vdd9" podStartSLOduration=3.346627553 podStartE2EDuration="3.346627553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:15:03.343172774 +0000 UTC m=+1522.090362278" watchObservedRunningTime="2026-10-31 23:15:03.346627553 +0000 UTC m=+1522.093817058"
Oct 31 23:15:03.469638 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:03.459314    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8" podStartSLOduration=3.459297116 podStartE2EDuration="3.459297116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:15:03.427789056 +0000 UTC m=+1522.174978561" watchObservedRunningTime="2026-10-31 23:15:03.459297116 +0000 UTC m=+1522.206486622"
Oct 31 23:15:03.515380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:03.513704    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b2c6553b-29891475-h4rm9" podStartSLOduration=3.513681224 podStartE2EDuration="3.513681224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:15:03.480693672 +0000 UTC m=+1522.227883180" watchObservedRunningTime="2026-10-31 23:15:03.513681224 +0000 UTC m=+1522.260870729"
Oct 31 23:15:04.390891 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:04.390854    4301 generic.go:356] "Generic (PLEG): container finished" podID="865bc4e8-2471-4a65-b11e-5ebc94b58fee" containerID="038f00ba7af0a0ee2169df172102f4fe513e8f4518cce01fccdbc49a188241a5" exitCode=60
Oct 31 23:15:04.391407 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:04.390992    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b2c6553b-29891475-h4rm9" event={"ID":"865bc4e8-2471-4a65-b11e-5ebc94b58fee","Type":"ContainerDied","Data":"038f00ba7af0a0ee2169df172102f4fe513e8f4518cce01fccdbc49a188241a5"}
Oct 31 23:15:04.391407 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:04.391038    4301 scope.go:122] "RemoveContainer" containerID="f09f447a3eae0b234fdf487ca9ab1cca761f23ad90b6913fbad0b9b503d82e80"
Oct 31 23:15:04.391567 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:04.391550    4301 scope.go:122] "RemoveContainer" containerID="038f00ba7af0a0ee2169df172102f4fe513e8f4518cce01fccdbc49a188241a5"
Oct 31 23:15:04.391909 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:15:04.391879    4301 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-b2c6553b-29891475-h4rm9_odh-ai-gateway-infra(865bc4e8-2471-4a65-b11e-5ebc94b58fee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b2c6553b-29891475-h4rm9" podUID="865bc4e8-2471-4a65-b11e-5ebc94b58fee"
Oct 31 23:15:04.397658 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:04.397613    4301 generic.go:356] "Generic (PLEG): container finished" podID="ce874bef-ee28-46ca-aacc-bb04f21486ea" containerID="d4a9bd6166532e79e924e4d9fdfa51015ab608f46cbf0bfbf8e7100478b9622b" exitCode=60
Oct 31 23:15:04.397802 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:04.397694    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8" event={"ID":"ce874bef-ee28-46ca-aacc-bb04f21486ea","Type":"ContainerDied","Data":"d4a9bd6166532e79e924e4d9fdfa51015ab608f46cbf0bfbf8e7100478b9622b"}
Oct 31 23:15:04.399165 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:04.399142    4301 scope.go:122] "RemoveContainer" containerID="d4a9bd6166532e79e924e4d9fdfa51015ab608f46cbf0bfbf8e7100478b9622b"
Oct 31 23:15:04.399605 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:15:04.399579    4301 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8_odh-ai-gateway-infra(ce874bef-ee28-46ca-aacc-bb04f21486ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8" podUID="ce874bef-ee28-46ca-aacc-bb04f21486ea"
Oct 31 23:15:04.402191 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:04.402004    4301 generic.go:356] "Generic (PLEG): container finished" podID="9c5d8c65-bbe8-486d-b24c-8f6a188f52c3" containerID="8119859baebf4034b1a2ca8b9104c18133bf695404b8c78929c3eae310b97dbf" exitCode=60
Oct 31 23:15:04.402191 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:04.402085    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg" event={"ID":"9c5d8c65-bbe8-486d-b24c-8f6a188f52c3","Type":"ContainerDied","Data":"8119859baebf4034b1a2ca8b9104c18133bf695404b8c78929c3eae310b97dbf"}
Oct 31 23:15:04.402789 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:04.402771    4301 scope.go:122] "RemoveContainer" containerID="8119859baebf4034b1a2ca8b9104c18133bf695404b8c78929c3eae310b97dbf"
Oct 31 23:15:04.403346 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:15:04.403322    4301 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg_odh-ai-gateway-infra(9c5d8c65-bbe8-486d-b24c-8f6a188f52c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg" podUID="9c5d8c65-bbe8-486d-b24c-8f6a188f52c3"
Oct 31 23:15:04.404939 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:04.404914    4301 generic.go:356] "Generic (PLEG): container finished" podID="09828acd-1f45-4f0b-b2df-dce7f38c2e3d" containerID="45bd8808074a3e3436a56c41e6ed526f6562e992ac2b5037d398b3449322a9cd" exitCode=60
Oct 31 23:15:04.405043 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:04.404993    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ed6117ae-29891475-7vdd9" event={"ID":"09828acd-1f45-4f0b-b2df-dce7f38c2e3d","Type":"ContainerDied","Data":"45bd8808074a3e3436a56c41e6ed526f6562e992ac2b5037d398b3449322a9cd"}
Oct 31 23:15:04.405337 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:04.405321    4301 scope.go:122] "RemoveContainer" containerID="45bd8808074a3e3436a56c41e6ed526f6562e992ac2b5037d398b3449322a9cd"
Oct 31 23:15:04.405767 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:15:04.405744    4301 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-ed6117ae-29891475-7vdd9_odh-ai-gateway-infra(09828acd-1f45-4f0b-b2df-dce7f38c2e3d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ed6117ae-29891475-7vdd9" podUID="09828acd-1f45-4f0b-b2df-dce7f38c2e3d"
Oct 31 23:15:04.416003 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:04.415975    4301 scope.go:122] "RemoveContainer" containerID="2e469015c9cc62cff665b7eab2ed713b53d39d5a0c56ef5c21dd112316f122e0"
Oct 31 23:15:04.431013 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:04.430988    4301 scope.go:122] "RemoveContainer" containerID="9c9a5eb3b0d2f7c57735eff2db1f3b9fb860a60c0698d362c020999b389655cb"
Oct 31 23:15:04.446912 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:04.446891    4301 scope.go:122] "RemoveContainer" containerID="5a54f187f5985988803a7f7beccac6063dbbee4e6449ca71994383d62c031bb7"
Oct 31 23:15:04.949056 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:04.949023    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:15:05.420177 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:05.420150    4301 scope.go:122] "RemoveContainer" containerID="d4a9bd6166532e79e924e4d9fdfa51015ab608f46cbf0bfbf8e7100478b9622b"
Oct 31 23:15:05.420679 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:15:05.420648    4301 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8_odh-ai-gateway-infra(ce874bef-ee28-46ca-aacc-bb04f21486ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8" podUID="ce874bef-ee28-46ca-aacc-bb04f21486ea"
Oct 31 23:15:05.421945 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:05.421924    4301 scope.go:122] "RemoveContainer" containerID="8119859baebf4034b1a2ca8b9104c18133bf695404b8c78929c3eae310b97dbf"
Oct 31 23:15:05.422507 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:15:05.422482    4301 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg_odh-ai-gateway-infra(9c5d8c65-bbe8-486d-b24c-8f6a188f52c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg" podUID="9c5d8c65-bbe8-486d-b24c-8f6a188f52c3"
Oct 31 23:15:05.423781 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:05.423764    4301 scope.go:122] "RemoveContainer" containerID="45bd8808074a3e3436a56c41e6ed526f6562e992ac2b5037d398b3449322a9cd"
Oct 31 23:15:05.424222 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:15:05.424200    4301 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-ed6117ae-29891475-7vdd9_odh-ai-gateway-infra(09828acd-1f45-4f0b-b2df-dce7f38c2e3d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ed6117ae-29891475-7vdd9" podUID="09828acd-1f45-4f0b-b2df-dce7f38c2e3d"
Oct 31 23:15:05.425432 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:05.425413    4301 scope.go:122] "RemoveContainer" containerID="038f00ba7af0a0ee2169df172102f4fe513e8f4518cce01fccdbc49a188241a5"
Oct 31 23:15:05.425677 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:15:05.425659    4301 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-b2c6553b-29891475-h4rm9_odh-ai-gateway-infra(865bc4e8-2471-4a65-b11e-5ebc94b58fee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b2c6553b-29891475-h4rm9" podUID="865bc4e8-2471-4a65-b11e-5ebc94b58fee"
Oct 31 23:15:05.745545 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:05.745445    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b2c6553b-29891475-h4rm9"]
Oct 31 23:15:05.890382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:05.884889    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4"]
Oct 31 23:15:05.890382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:05.885086    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4" podUID="72a95a93-f42b-4829-aaf5-b388a7482e8e" containerName="maas-api" containerID="cri-o://51fc005f0ef0c2d14e8e185999bef18f506c7d850e4a71cd6dea0e7247b47e54" gracePeriod=30
Oct 31 23:15:05.937332 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:05.937276    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ed6117ae-29891475-7vdd9"]
Oct 31 23:15:06.816940 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:06.815911    4301 generic.go:356] "Generic (PLEG): container finished" podID="72a95a93-f42b-4829-aaf5-b388a7482e8e" containerID="51fc005f0ef0c2d14e8e185999bef18f506c7d850e4a71cd6dea0e7247b47e54" exitCode=0
Oct 31 23:15:06.816940 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:06.816436    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4" event={"ID":"72a95a93-f42b-4829-aaf5-b388a7482e8e","Type":"ContainerDied","Data":"51fc005f0ef0c2d14e8e185999bef18f506c7d850e4a71cd6dea0e7247b47e54"}
Oct 31 23:15:06.828163 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:06.828115    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"]
Oct 31 23:15:06.830150 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:06.830124    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76" podUID="3f037628-d51c-48e0-be4e-5d9c99a10332" containerName="istio-proxy" containerID="cri-o://be200d967c5ee63551ff33339c18d452825ab9f9a34a5f6752689f4b9bc81cd0" gracePeriod=30
Oct 31 23:15:06.853922 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:06.853831    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4"
Oct 31 23:15:06.858479 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:06.856295    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72a95a93-f42b-4829-aaf5-b388a7482e8e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72a95a93-f42b-4829-aaf5-b388a7482e8e-maas-api-tls\") pod \"72a95a93-f42b-4829-aaf5-b388a7482e8e\" (UID: \"72a95a93-f42b-4829-aaf5-b388a7482e8e\") "
Oct 31 23:15:06.858479 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:06.856491    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72a95a93-f42b-4829-aaf5-b388a7482e8e-kube-api-access-zlhqz\" (UniqueName: \"kubernetes.io/projected/72a95a93-f42b-4829-aaf5-b388a7482e8e-kube-api-access-zlhqz\") pod \"72a95a93-f42b-4829-aaf5-b388a7482e8e\" (UID: \"72a95a93-f42b-4829-aaf5-b388a7482e8e\") "
Oct 31 23:15:06.858479 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:06.856560    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72a95a93-f42b-4829-aaf5-b388a7482e8e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72a95a93-f42b-4829-aaf5-b388a7482e8e-openshift-service-ca\") pod \"72a95a93-f42b-4829-aaf5-b388a7482e8e\" (UID: \"72a95a93-f42b-4829-aaf5-b388a7482e8e\") "
Oct 31 23:15:06.858479 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:06.856649    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72a95a93-f42b-4829-aaf5-b388a7482e8e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/72a95a93-f42b-4829-aaf5-b388a7482e8e-metrics-certs\") pod \"72a95a93-f42b-4829-aaf5-b388a7482e8e\" (UID: \"72a95a93-f42b-4829-aaf5-b388a7482e8e\") "
Oct 31 23:15:06.867839 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:06.867811    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a95a93-f42b-4829-aaf5-b388a7482e8e-openshift-service-ca" pod "72a95a93-f42b-4829-aaf5-b388a7482e8e" (UID: "72a95a93-f42b-4829-aaf5-b388a7482e8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:15:07.050497 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:07.050286    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72a95a93-f42b-4829-aaf5-b388a7482e8e-maas-api-tls" pod "72a95a93-f42b-4829-aaf5-b388a7482e8e" (UID: "72a95a93-f42b-4829-aaf5-b388a7482e8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:15:07.070376 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:07.068880    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72a95a93-f42b-4829-aaf5-b388a7482e8e-maas-api-tls\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:15:07.070376 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:07.068916    4301 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72a95a93-f42b-4829-aaf5-b388a7482e8e-openshift-service-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:15:07.074811 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:07.074771    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72a95a93-f42b-4829-aaf5-b388a7482e8e-metrics-certs" pod "72a95a93-f42b-4829-aaf5-b388a7482e8e" (UID: "72a95a93-f42b-4829-aaf5-b388a7482e8e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:15:07.123463 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:07.123351    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a95a93-f42b-4829-aaf5-b388a7482e8e-kube-api-access-zlhqz" pod "72a95a93-f42b-4829-aaf5-b388a7482e8e" (UID: "72a95a93-f42b-4829-aaf5-b388a7482e8e"). InnerVolumeSpecName "kube-api-access-zlhqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:15:07.194381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:07.187834    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlhqz\" (UniqueName: \"kubernetes.io/projected/72a95a93-f42b-4829-aaf5-b388a7482e8e-kube-api-access-zlhqz\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:15:07.194381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:07.187871    4301 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72a95a93-f42b-4829-aaf5-b388a7482e8e-metrics-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:15:07.519378 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:07.507525    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:07.560394 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:07.554449    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:07.560394 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:07.554806    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Oct 31 23:15:07.560394 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:07.554846    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 23:15:07.623680 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:07.604830    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": context deadline exceeded"
Oct 31 23:15:07.728399 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:07.727439    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:07.862001 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:07.861959    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4" event={"ID":"72a95a93-f42b-4829-aaf5-b388a7482e8e","Type":"ContainerDied","Data":"b3d55fc6c877a1a6427fd92d4aa7b9d3fac3325d31db29eacb3322752c794584"}
Oct 31 23:15:07.862615 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:07.862594    4301 scope.go:122] "RemoveContainer" containerID="51fc005f0ef0c2d14e8e185999bef18f506c7d850e4a71cd6dea0e7247b47e54"
Oct 31 23:15:07.862860 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:07.862843    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4"
Oct 31 23:15:07.944251 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:07.943557    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4"]
Oct 31 23:15:07.944251 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:07.943818    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 31 23:15:08.159644 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.157678    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-ed6117ae-69dd446956-qls9z" podUID="4327b4bd-bb08-42da-933e-3e332b2eb85a" containerName="payload-processing" containerID="cri-o://85f0f53d3d8bd6d0505bdf4e20e55545ea0171a1439f2a874d9c1039fc2ed5d4" gracePeriod=25
Oct 31 23:15:08.177946 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.162792    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ed6117ae-6746cb564-bgvn4"]
Oct 31 23:15:08.364650 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.322719    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"]
Oct 31 23:15:08.364650 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.323753    4301 cpu_manager.go:414] "containerMap: removing container" podUID="72a95a93-f42b-4829-aaf5-b388a7482e8e" containerName="maas-api"
Oct 31 23:15:08.364650 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.323774    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72a95a93-f42b-4829-aaf5-b388a7482e8e" containerName="maas-api"
Oct 31 23:15:08.364650 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.323953    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="72a95a93-f42b-4829-aaf5-b388a7482e8e" containerName="maas-api"
Oct 31 23:15:08.364650 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.329821    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:15:08.426396 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.392105    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-istio-envoy\") pod \"e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") " pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:15:08.426396 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.392164    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92fe8257-17fc-4467-9287-b867a88a4f95-istio-token\") pod \"e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") " pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:15:08.426396 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.392257    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzwmx\" (UniqueName: \"kubernetes.io/projected/92fe8257-17fc-4467-9287-b867a88a4f95-kube-api-access-fzwmx\") pod \"e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") " pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:15:08.426396 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.392371    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92fe8257-17fc-4467-9287-b867a88a4f95-istio-podinfo\") pod \"e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") " pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:15:08.426396 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.392545    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-credential-socket\") pod \"e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") " pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:15:08.426396 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.392599    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-istio-data\") pod \"e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") " pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:15:08.426396 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.392646    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-workload-certs\") pod \"e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") " pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:15:08.426396 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.392671    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-workload-socket\") pod \"e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") " pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:15:08.426396 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.392731    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92fe8257-17fc-4467-9287-b867a88a4f95-istiod-ca-cert\") pod \"e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") " pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:15:08.426396 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.417033    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-56b15fa9-openshift-default-dockercfg-pb8lc\""
Oct 31 23:15:08.502386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.499665    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-istio-envoy\") pod \"e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") " pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:15:08.502386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.499723    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92fe8257-17fc-4467-9287-b867a88a4f95-istio-token\") pod \"e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") " pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:15:08.502386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.499835    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzwmx\" (UniqueName: \"kubernetes.io/projected/92fe8257-17fc-4467-9287-b867a88a4f95-kube-api-access-fzwmx\") pod \"e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") " pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:15:08.502386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.499956    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92fe8257-17fc-4467-9287-b867a88a4f95-istio-podinfo\") pod \"e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") " pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:15:08.502386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.500033    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-credential-socket\") pod \"e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") " pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:15:08.502386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.500104    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-istio-data\") pod \"e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") " pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:15:08.502386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.500159    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-workload-certs\") pod \"e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") " pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:15:08.502386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.500186    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-workload-socket\") pod \"e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") " pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:15:08.502386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.500265    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92fe8257-17fc-4467-9287-b867a88a4f95-istiod-ca-cert\") pod \"e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") " pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:15:08.502386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.501126    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92fe8257-17fc-4467-9287-b867a88a4f95-istiod-ca-cert\") pod \"e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") " pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:15:08.504304 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.503529    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-istio-data\") pod \"e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") " pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:15:08.504304 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.503790    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-credential-socket\") pod \"e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") " pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:15:08.504304 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.504035    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-workload-certs\") pod \"e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") " pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:15:08.505016 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.504970    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-workload-socket\") pod \"e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") " pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:15:08.630514 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.572237    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:08.630514 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.585652    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-istio-envoy\") pod \"e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") " pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:15:08.630514 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.610264    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"]
Oct 31 23:15:08.630514 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.619946    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92fe8257-17fc-4467-9287-b867a88a4f95-istio-token\") pod \"e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") " pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:15:08.657758 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.657726    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzwmx\" (UniqueName: \"kubernetes.io/projected/92fe8257-17fc-4467-9287-b867a88a4f95-kube-api-access-fzwmx\") pod \"e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") " pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:15:08.704614 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.704505    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:08.763386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.747290    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:08.763386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.755648    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:15:08.848393 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.846894    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92fe8257-17fc-4467-9287-b867a88a4f95-istio-podinfo\") pod \"e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") " pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:15:08.990562 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:08.990470    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:09.035384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:09.031959    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:15:09.066400 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:09.056647    4301 generic.go:356] "Generic (PLEG): container finished" podID="4327b4bd-bb08-42da-933e-3e332b2eb85a" containerID="85f0f53d3d8bd6d0505bdf4e20e55545ea0171a1439f2a874d9c1039fc2ed5d4" exitCode=0
Oct 31 23:15:09.066400 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:09.056881    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ed6117ae-69dd446956-qls9z" event={"ID":"4327b4bd-bb08-42da-933e-3e332b2eb85a","Type":"ContainerDied","Data":"85f0f53d3d8bd6d0505bdf4e20e55545ea0171a1439f2a874d9c1039fc2ed5d4"}
Oct 31 23:15:09.111434 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:09.096592    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b2c6553b-29891475-h4rm9"
Oct 31 23:15:09.124707 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:09.124673    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:15:09.267712 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:09.224652    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/865bc4e8-2471-4a65-b11e-5ebc94b58fee-kube-api-access-tcv9n\" (UniqueName: \"kubernetes.io/projected/865bc4e8-2471-4a65-b11e-5ebc94b58fee-kube-api-access-tcv9n\") pod \"865bc4e8-2471-4a65-b11e-5ebc94b58fee\" (UID: \"865bc4e8-2471-4a65-b11e-5ebc94b58fee\") "
Oct 31 23:15:09.267712 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:09.260088    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/865bc4e8-2471-4a65-b11e-5ebc94b58fee-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/865bc4e8-2471-4a65-b11e-5ebc94b58fee-maas-api-service-ca\") pod \"865bc4e8-2471-4a65-b11e-5ebc94b58fee\" (UID: \"865bc4e8-2471-4a65-b11e-5ebc94b58fee\") "
Oct 31 23:15:09.360628 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:09.335139    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 23:15:09.360628 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:09.355856    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/865bc4e8-2471-4a65-b11e-5ebc94b58fee-maas-api-service-ca" pod "865bc4e8-2471-4a65-b11e-5ebc94b58fee" (UID: "865bc4e8-2471-4a65-b11e-5ebc94b58fee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:15:09.500400 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:09.498223    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:09.500400 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:09.498609    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:09.553395 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:09.552445    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/865bc4e8-2471-4a65-b11e-5ebc94b58fee-maas-api-service-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:15:09.586619 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:09.569177    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72a95a93-f42b-4829-aaf5-b388a7482e8e" path="/var/lib/kubelet/pods/72a95a93-f42b-4829-aaf5-b388a7482e8e/volumes"
Oct 31 23:15:09.621686 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:09.593666    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:09.621686 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:09.616448    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:09.712390 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:09.630586    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/865bc4e8-2471-4a65-b11e-5ebc94b58fee-kube-api-access-tcv9n" pod "865bc4e8-2471-4a65-b11e-5ebc94b58fee" (UID: "865bc4e8-2471-4a65-b11e-5ebc94b58fee"). InnerVolumeSpecName "kube-api-access-tcv9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:15:09.712390 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:09.658975    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcv9n\" (UniqueName: \"kubernetes.io/projected/865bc4e8-2471-4a65-b11e-5ebc94b58fee-kube-api-access-tcv9n\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:15:09.747708 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:09.747622    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:09.853762 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:09.751562    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:09.855320 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:09.855285    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:15:10.122449 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:10.036759    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:10.168992 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:10.168955    4301 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 23:15:10.196482 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:10.196433    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz" podUID="b0f47424-1492-437c-9da2-0712ea4266b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:10.332459 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:10.318032    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:10.356389 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:10.343477    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:10.356389 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:10.344078    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:15:10.418633 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:10.418548    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b2c6553b-29891475-h4rm9"
Oct 31 23:15:10.419012 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:10.418985    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b2c6553b-29891475-h4rm9" event={"ID":"865bc4e8-2471-4a65-b11e-5ebc94b58fee","Type":"ContainerDied","Data":"7c53c91b46e42899406abe6c050255079b2613c39e507aa2bd56539c175f1b6b"}
Oct 31 23:15:10.419194 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:10.419179    4301 scope.go:122] "RemoveContainer" containerID="038f00ba7af0a0ee2169df172102f4fe513e8f4518cce01fccdbc49a188241a5"
Oct 31 23:15:10.506523 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:10.506475    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:10.507711 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:10.507674    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:10.508474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:10.508455    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:15:10.536731 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:10.536684    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:10.537588 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:10.537557    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:15:10.562121 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:10.561201    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:10.621517 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:10.621472    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 23:15:10.622083 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:10.622057    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 23:15:10.622524 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:10.622505    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:15:10.622729 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:10.622708    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:15:11.067677 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:11.067633    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 23:15:11.125731 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:11.125677    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:11.134762 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:11.134727    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:11.140515 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:11.140480    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": context deadline exceeded"
Oct 31 23:15:11.206201 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:11.206166    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ed6117ae-29891475-7vdd9"
Oct 31 23:15:11.342384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:11.337467    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 23:15:11.342384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:11.338227    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:15:11.365903 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:11.365851    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:11.419006 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:11.374373    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09828acd-1f45-4f0b-b2df-dce7f38c2e3d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09828acd-1f45-4f0b-b2df-dce7f38c2e3d-maas-api-service-ca\") pod \"09828acd-1f45-4f0b-b2df-dce7f38c2e3d\" (UID: \"09828acd-1f45-4f0b-b2df-dce7f38c2e3d\") "
Oct 31 23:15:11.419006 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:11.374697    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09828acd-1f45-4f0b-b2df-dce7f38c2e3d-kube-api-access-dng2l\" (UniqueName: \"kubernetes.io/projected/09828acd-1f45-4f0b-b2df-dce7f38c2e3d-kube-api-access-dng2l\") pod \"09828acd-1f45-4f0b-b2df-dce7f38c2e3d\" (UID: \"09828acd-1f45-4f0b-b2df-dce7f38c2e3d\") "
Oct 31 23:15:11.419006 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:11.405536    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09828acd-1f45-4f0b-b2df-dce7f38c2e3d-maas-api-service-ca" pod "09828acd-1f45-4f0b-b2df-dce7f38c2e3d" (UID: "09828acd-1f45-4f0b-b2df-dce7f38c2e3d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:15:11.598942 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:11.595553    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:11.608871 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:11.595652    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 31 23:15:11.797526 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:11.795618    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09828acd-1f45-4f0b-b2df-dce7f38c2e3d-kube-api-access-dng2l" pod "09828acd-1f45-4f0b-b2df-dce7f38c2e3d" (UID: "09828acd-1f45-4f0b-b2df-dce7f38c2e3d"). InnerVolumeSpecName "kube-api-access-dng2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:15:11.797526 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:11.796129    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:11.797526 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:11.796535    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 23:15:11.833843 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:11.833813    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09828acd-1f45-4f0b-b2df-dce7f38c2e3d-maas-api-service-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:15:11.930463 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:11.834084    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:12.030983 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:12.012332    4301 patch_prober.go:29] interesting pod/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 31 23:15:12.031256 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:12.031219    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76" podUID="3f037628-d51c-48e0-be4e-5d9c99a10332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 31 23:15:12.034075 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:12.034032    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dng2l\" (UniqueName: \"kubernetes.io/projected/09828acd-1f45-4f0b-b2df-dce7f38c2e3d-kube-api-access-dng2l\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:15:12.090966 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:12.090912    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:12.142572 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:12.142516    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 31 23:15:12.220719 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:12.220658    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:12.395389 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:12.371528    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 23:15:12.480398 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:12.469651    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 23:15:12.608536 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:12.608476    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:12.609272 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:15:12.609249    4301 logging.go:55] [core] [Channel #161 SubChannel #162]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.73:50052", ServerName: "10.133.0.73:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.73:50052: connect: connection refused"
Oct 31 23:15:12.633484 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:12.609488    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:12.807539 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:12.807423    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 23:15:12.832608 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:12.832556    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 23:15:12.853439 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:15:12.853393    4301 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.454s"
Oct 31 23:15:12.862416 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:12.862387    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:15:13.002147 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:13.002103    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 23:15:13.084459 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:13.072112    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ed6117ae-29891475-7vdd9" event={"ID":"09828acd-1f45-4f0b-b2df-dce7f38c2e3d","Type":"ContainerDied","Data":"2f307cad85e080f684e6d54cdb70034339491d7b68e78df93590c25b28d53d2d"}
Oct 31 23:15:13.084459 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:13.072212    4301 scope.go:122] "RemoveContainer" containerID="45bd8808074a3e3436a56c41e6ed526f6562e992ac2b5037d398b3449322a9cd"
Oct 31 23:15:13.084459 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:13.072553    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ed6117ae-29891475-7vdd9"
Oct 31 23:15:13.199204 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:13.194756    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:13.227341 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:13.209257    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 23:15:13.227341 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:13.210112    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-migrate-ed6117ae-69dd446956-qls9z" podUID="4327b4bd-bb08-42da-933e-3e332b2eb85a" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.73:50052\" within 1s: context deadline exceeded"
Oct 31 23:15:13.260453 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:13.248535    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 31 23:15:13.558021 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:13.553229    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:15:13.558021 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:13.553344    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:15:13.558021 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:13.553419    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:15:13.639572 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:13.623179    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:15:13.740433 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:13.719330    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:13.814783 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:13.814738    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:15:13.966438 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:13.845521    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 23:15:13.966438 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:13.860632    4301 scope.go:122] "RemoveContainer" containerID="d4a9bd6166532e79e924e4d9fdfa51015ab608f46cbf0bfbf8e7100478b9622b"
Oct 31 23:15:13.966438 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:13.879610    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 23:15:13.966438 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:13.934451    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:13.995961 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:13.995263    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:14.034673 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:14.034629    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:14.111761 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:14.111721    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:14.264265 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:14.264228    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 31 23:15:14.566435 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:14.552503    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:14.832119 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:14.832008    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:14.889262 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:14.849549    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 23:15:14.900891 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:14.900845    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ed6117ae-69dd446956-qls9z"
Oct 31 23:15:14.901718 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:14.880272    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 23:15:14.928115 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:15:14.928044    4301 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.059s"
Oct 31 23:15:14.928598 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:14.928548    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb" event={"ID":"92fe8257-17fc-4467-9287-b867a88a4f95","Type":"ContainerStarted","Data":"b632d1a059c8346b2c48a578d20ec4b3c0526c2f94ffca5403c078cc4e30e590"}
Oct 31 23:15:14.928843 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:14.928827    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:15:14.928988 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:14.928978    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:15:15.039606 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:15.039563    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:15.143683 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:15:15.135153    4301 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09828acd_1f45_4f0b_b2df_dce7f38c2e3d.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 23:15:15.232400 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:15.230834    4301 generic.go:356] "Generic (PLEG): container finished" podID="3f037628-d51c-48e0-be4e-5d9c99a10332" containerID="be200d967c5ee63551ff33339c18d452825ab9f9a34a5f6752689f4b9bc81cd0" exitCode=0
Oct 31 23:15:15.232400 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:15.230968    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76" event={"ID":"3f037628-d51c-48e0-be4e-5d9c99a10332","Type":"ContainerDied","Data":"be200d967c5ee63551ff33339c18d452825ab9f9a34a5f6752689f4b9bc81cd0"}
Oct 31 23:15:15.395648 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:15.395555    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4327b4bd-bb08-42da-933e-3e332b2eb85a-kube-api-access-w54ff\" (UniqueName: \"kubernetes.io/projected/4327b4bd-bb08-42da-933e-3e332b2eb85a-kube-api-access-w54ff\") pod \"4327b4bd-bb08-42da-933e-3e332b2eb85a\" (UID: \"4327b4bd-bb08-42da-933e-3e332b2eb85a\") "
Oct 31 23:15:15.597806 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:15.578964    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4327b4bd-bb08-42da-933e-3e332b2eb85a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4327b4bd-bb08-42da-933e-3e332b2eb85a-plugins-config-volume\") pod \"4327b4bd-bb08-42da-933e-3e332b2eb85a\" (UID: \"4327b4bd-bb08-42da-933e-3e332b2eb85a\") "
Oct 31 23:15:15.597806 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:15.402350    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:15.601578 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:15.422042    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:15.619182 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:15.619150    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ed6117ae-69dd446956-qls9z"
Oct 31 23:15:15.656621 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:15:15.422560    4301 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09828acd_1f45_4f0b_b2df_dce7f38c2e3d.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 23:15:15.670014 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:15.602626    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4327b4bd-bb08-42da-933e-3e332b2eb85a-plugins-config-volume" pod "4327b4bd-bb08-42da-933e-3e332b2eb85a" (UID: "4327b4bd-bb08-42da-933e-3e332b2eb85a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:15:15.670014 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:15.658246    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4327b4bd-bb08-42da-933e-3e332b2eb85a-kube-api-access-w54ff" pod "4327b4bd-bb08-42da-933e-3e332b2eb85a" (UID: "4327b4bd-bb08-42da-933e-3e332b2eb85a"). InnerVolumeSpecName "kube-api-access-w54ff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:15:15.670259 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:15.670190    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 31 23:15:15.797391 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:15.795332    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w54ff\" (UniqueName: \"kubernetes.io/projected/4327b4bd-bb08-42da-933e-3e332b2eb85a-kube-api-access-w54ff\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:15:15.797391 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:15.795382    4301 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4327b4bd-bb08-42da-933e-3e332b2eb85a-plugins-config-volume\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:15:15.921820 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:15.921716    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 31 23:15:15.969858 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:15.969794    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:15.971872 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:15.971834    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 23:15:16.016605 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:16.016558    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:16.267429 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:16.262464    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:16.604234 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:16.604185    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:16.625519 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:16.625468    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:16.629119 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:16.629082    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 31 23:15:16.750283 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:16.750237    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:16.944468 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:16.934567    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:16.955125 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:16.955069    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:17.012009 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:15:17.011973    4301 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.54s"
Oct 31 23:15:17.015397 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:17.012574    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:15:17.015624 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:17.015598    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ed6117ae-69dd446956-qls9z" event={"ID":"4327b4bd-bb08-42da-933e-3e332b2eb85a","Type":"ContainerDied","Data":"9540e55b69d2b2f2d550ca6b0e47219bae69c5aff7b6da5d0731a1f0b35ec3e0"}
Oct 31 23:15:17.015764 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:17.015748    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ac5ff20f-6fb5d9b-f9qlx"]
Oct 31 23:15:17.015931 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:17.015917    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:15:17.016104 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:17.016091    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ac5ff20f-6567b4b69f-sgrvt"]
Oct 31 23:15:17.033579 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:17.033552    4301 scope.go:122] "RemoveContainer" containerID="85f0f53d3d8bd6d0505bdf4e20e55545ea0171a1439f2a874d9c1039fc2ed5d4"
Oct 31 23:15:17.035587 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:17.035566    4301 scope.go:122] "RemoveContainer" containerID="8119859baebf4034b1a2ca8b9104c18133bf695404b8c78929c3eae310b97dbf"
Oct 31 23:15:17.097349 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:17.086290    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b2c6553b-29891475-h4rm9"]
Oct 31 23:15:17.097349 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:17.086344    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-b2c6553b-29891475-h4rm9"]
Oct 31 23:15:17.097349 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:17.088461    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded"
Oct 31 23:15:17.097349 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:17.088540    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 23:15:17.097349 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:17.088853    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 23:15:17.100502 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:17.100380    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"]
Oct 31 23:15:17.127310 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:17.123666    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ed6117ae-29891475-7vdd9"]
Oct 31 23:15:17.165113 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:17.164992    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ed6117ae-29891475-7vdd9"]
Oct 31 23:15:17.317779 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:17.303149    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:17.317779 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:17.303670    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:15:17.357388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:17.340628    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:17.358181 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:17.358113    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:17.616008 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:17.615476    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:17.649436 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:17.649398    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:17.671953 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:17.658965    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 23:15:17.671953 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:17.670263    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:17.698964 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:17.698927    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09828acd-1f45-4f0b-b2df-dce7f38c2e3d" path="/var/lib/kubelet/pods/09828acd-1f45-4f0b-b2df-dce7f38c2e3d/volumes"
Oct 31 23:15:17.701581 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:17.699823    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="865bc4e8-2471-4a65-b11e-5ebc94b58fee" path="/var/lib/kubelet/pods/865bc4e8-2471-4a65-b11e-5ebc94b58fee/volumes"
Oct 31 23:15:17.702427 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:17.702224    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:15:17.823968 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:17.820467    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 31 23:15:17.956421 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:17.952612    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": context deadline exceeded"
Oct 31 23:15:18.009440 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:17.966300    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:18.009440 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:17.967416    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:15:18.009440 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:17.988025    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:15:18.025847 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.025814    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb" event={"ID":"92fe8257-17fc-4467-9287-b867a88a4f95","Type":"ContainerStarted","Data":"51afe40fc53888cd8c33e3c5e8968fc3227746e51c71594ef958ce6264ea867c"}
Oct 31 23:15:18.029839 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.029815    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:15:18.030861 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.030837    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76" event={"ID":"3f037628-d51c-48e0-be4e-5d9c99a10332","Type":"ContainerDied","Data":"042b69f3e079a7911786bfedaa0f2581afe8fbaad6e018fb6d3ff968bc3c8069"}
Oct 31 23:15:18.031055 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.031038    4301 scope.go:122] "RemoveContainer" containerID="be200d967c5ee63551ff33339c18d452825ab9f9a34a5f6752689f4b9bc81cd0"
Oct 31 23:15:18.033911 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.032283    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:18.038864 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.038844    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:15:18.041836 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.041814    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8" event={"ID":"ce874bef-ee28-46ca-aacc-bb04f21486ea","Type":"ContainerStarted","Data":"8d35956df007b8a3010b6b55a5c07b6ec56cd9c4d4a604e88c07814cc6448ca6"}
Oct 31 23:15:18.045064 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.045046    4301 scope.go:122] "RemoveContainer" containerID="8d35956df007b8a3010b6b55a5c07b6ec56cd9c4d4a604e88c07814cc6448ca6"
Oct 31 23:15:18.047378 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:15:18.047340    4301 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8_odh-ai-gateway-infra(ce874bef-ee28-46ca-aacc-bb04f21486ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8" podUID="ce874bef-ee28-46ca-aacc-bb04f21486ea"
Oct 31 23:15:18.065208 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.060961    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:15:18.093448 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.092281    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:18.134615 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.134584    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:15:18.139570 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.135681    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-token\" (UniqueName: \"kubernetes.io/projected/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-token\") pod \"3f037628-d51c-48e0-be4e-5d9c99a10332\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") "
Oct 31 23:15:18.242438 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.241519    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f037628-d51c-48e0-be4e-5d9c99a10332-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f037628-d51c-48e0-be4e-5d9c99a10332-istiod-ca-cert\") pod \"3f037628-d51c-48e0-be4e-5d9c99a10332\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") "
Oct 31 23:15:18.242438 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.241573    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-data\") pod \"3f037628-d51c-48e0-be4e-5d9c99a10332\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") "
Oct 31 23:15:18.242438 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.241703    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-workload-certs\") pod \"3f037628-d51c-48e0-be4e-5d9c99a10332\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") "
Oct 31 23:15:18.242438 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.241766    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f037628-d51c-48e0-be4e-5d9c99a10332-kube-api-access-pjqcz\" (UniqueName: \"kubernetes.io/projected/3f037628-d51c-48e0-be4e-5d9c99a10332-kube-api-access-pjqcz\") pod \"3f037628-d51c-48e0-be4e-5d9c99a10332\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") "
Oct 31 23:15:18.242438 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.241818    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-credential-socket\") pod \"3f037628-d51c-48e0-be4e-5d9c99a10332\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") "
Oct 31 23:15:18.242438 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.241956    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-podinfo\") pod \"3f037628-d51c-48e0-be4e-5d9c99a10332\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") "
Oct 31 23:15:18.242438 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.241994    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-envoy\") pod \"3f037628-d51c-48e0-be4e-5d9c99a10332\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") "
Oct 31 23:15:18.242438 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.242047    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-workload-socket\") pod \"3f037628-d51c-48e0-be4e-5d9c99a10332\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") "
Oct 31 23:15:18.250292 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.161428    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ed6117ae-69dd446956-qls9z"]
Oct 31 23:15:18.250292 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.249459    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:15:18.250292 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.249561    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:15:18.263651 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.263616    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-data" pod "3f037628-d51c-48e0-be4e-5d9c99a10332" (UID: "3f037628-d51c-48e0-be4e-5d9c99a10332"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:15:18.264101 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.264057    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f037628-d51c-48e0-be4e-5d9c99a10332-istiod-ca-cert" pod "3f037628-d51c-48e0-be4e-5d9c99a10332" (UID: "3f037628-d51c-48e0-be4e-5d9c99a10332"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:15:18.278713 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.278682    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-credential-socket" pod "3f037628-d51c-48e0-be4e-5d9c99a10332" (UID: "3f037628-d51c-48e0-be4e-5d9c99a10332"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:15:18.286634 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.286590    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-workload-certs" pod "3f037628-d51c-48e0-be4e-5d9c99a10332" (UID: "3f037628-d51c-48e0-be4e-5d9c99a10332"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:15:18.290606 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.290566    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-workload-socket" pod "3f037628-d51c-48e0-be4e-5d9c99a10332" (UID: "3f037628-d51c-48e0-be4e-5d9c99a10332"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:15:18.301792 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.301685    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f037628-d51c-48e0-be4e-5d9c99a10332-kube-api-access-pjqcz" pod "3f037628-d51c-48e0-be4e-5d9c99a10332" (UID: "3f037628-d51c-48e0-be4e-5d9c99a10332"). InnerVolumeSpecName "kube-api-access-pjqcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:15:18.305478 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.305441    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:18.308851 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.308817    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:15:18.341073 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.334649    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:15:18.631687 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.631645    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-envoy" pod "3f037628-d51c-48e0-be4e-5d9c99a10332" (UID: "3f037628-d51c-48e0-be4e-5d9c99a10332"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:15:18.647440 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.647344    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-token" pod "3f037628-d51c-48e0-be4e-5d9c99a10332" (UID: "3f037628-d51c-48e0-be4e-5d9c99a10332"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:15:18.650243 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.650197    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:18.651045 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.651016    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ed6117ae-69dd446956-qls9z"]
Oct 31 23:15:18.651661 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.651571    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-podinfo" pod "3f037628-d51c-48e0-be4e-5d9c99a10332" (UID: "3f037628-d51c-48e0-be4e-5d9c99a10332"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:15:18.651914 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.651894    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-podinfo\") pod \"3f037628-d51c-48e0-be4e-5d9c99a10332\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") "
Oct 31 23:15:18.652057 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.652044    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-envoy\") pod \"3f037628-d51c-48e0-be4e-5d9c99a10332\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") "
Oct 31 23:15:18.652301 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.652288    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-token\" (UniqueName: \"kubernetes.io/projected/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-token\") pod \"3f037628-d51c-48e0-be4e-5d9c99a10332\" (UID: \"3f037628-d51c-48e0-be4e-5d9c99a10332\") "
Oct 31 23:15:18.653902 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.653875    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:15:18.654059 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.654045    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjqcz\" (UniqueName: \"kubernetes.io/projected/3f037628-d51c-48e0-be4e-5d9c99a10332-kube-api-access-pjqcz\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:15:18.654148 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.654137    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:15:18.654238 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.654227    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:15:18.654316 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.654305    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f037628-d51c-48e0-be4e-5d9c99a10332-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:15:18.654420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.654406    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:15:18.655177 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:15:18.655160    4301 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/3f037628-d51c-48e0-be4e-5d9c99a10332/volumes/kubernetes.io~downward-api/istio-podinfo
Oct 31 23:15:18.655297 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.655282    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-podinfo" pod "3f037628-d51c-48e0-be4e-5d9c99a10332" (UID: "3f037628-d51c-48e0-be4e-5d9c99a10332"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:15:18.655449 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:15:18.655438    4301 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/3f037628-d51c-48e0-be4e-5d9c99a10332/volumes/kubernetes.io~empty-dir/istio-envoy
Oct 31 23:15:18.655550 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.655529    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-envoy" pod "3f037628-d51c-48e0-be4e-5d9c99a10332" (UID: "3f037628-d51c-48e0-be4e-5d9c99a10332"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:15:18.655668 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:15:18.655660    4301 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/3f037628-d51c-48e0-be4e-5d9c99a10332/volumes/kubernetes.io~projected/istio-token
Oct 31 23:15:18.655776 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.655764    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-token" pod "3f037628-d51c-48e0-be4e-5d9c99a10332" (UID: "3f037628-d51c-48e0-be4e-5d9c99a10332"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:15:18.659319 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.659299    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:15:18.676746 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.668482    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:15:18.690679 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.690655    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:15:18.716278 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.716227    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb" podStartSLOduration=10.716211955 podStartE2EDuration="10.716211955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:15:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:15:18.130263071 +0000 UTC m=+1536.877452577" watchObservedRunningTime="2026-10-31 23:15:18.716211955 +0000 UTC m=+1537.463401459"
Oct 31 23:15:18.765422 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.764750    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:15:18.765422 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.764781    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:15:18.765422 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:18.764800    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f037628-d51c-48e0-be4e-5d9c99a10332-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:15:19.048294 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:19.048262    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"
Oct 31 23:15:19.051513 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:19.051488    4301 generic.go:356] "Generic (PLEG): container finished" podID="ce874bef-ee28-46ca-aacc-bb04f21486ea" containerID="8d35956df007b8a3010b6b55a5c07b6ec56cd9c4d4a604e88c07814cc6448ca6" exitCode=60
Oct 31 23:15:19.051654 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:19.051561    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8" event={"ID":"ce874bef-ee28-46ca-aacc-bb04f21486ea","Type":"ContainerDied","Data":"8d35956df007b8a3010b6b55a5c07b6ec56cd9c4d4a604e88c07814cc6448ca6"}
Oct 31 23:15:19.051654 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:19.051623    4301 scope.go:122] "RemoveContainer" containerID="d4a9bd6166532e79e924e4d9fdfa51015ab608f46cbf0bfbf8e7100478b9622b"
Oct 31 23:15:19.052161 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:19.052129    4301 scope.go:122] "RemoveContainer" containerID="8d35956df007b8a3010b6b55a5c07b6ec56cd9c4d4a604e88c07814cc6448ca6"
Oct 31 23:15:19.052580 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:15:19.052554    4301 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8_odh-ai-gateway-infra(ce874bef-ee28-46ca-aacc-bb04f21486ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8" podUID="ce874bef-ee28-46ca-aacc-bb04f21486ea"
Oct 31 23:15:19.054292 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:19.054270    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg" event={"ID":"9c5d8c65-bbe8-486d-b24c-8f6a188f52c3","Type":"ContainerStarted","Data":"0e2af783791d01bacd341053ea1d7073b473c973aa7fcc7f1a971038ceea0881"}
Oct 31 23:15:19.055035 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:19.055010    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:15:19.055568 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:19.055547    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:15:19.108483 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:19.108449    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"]
Oct 31 23:15:19.118486 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:19.118450    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-ed6117ae-gw-openshift-default-56b9c8c78f-4tw76"]
Oct 31 23:15:19.206694 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:19.206664    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8"]
Oct 31 23:15:19.408028 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:19.407996    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f037628-d51c-48e0-be4e-5d9c99a10332" path="/var/lib/kubelet/pods/3f037628-d51c-48e0-be4e-5d9c99a10332/volumes"
Oct 31 23:15:19.408527 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:19.408511    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4327b4bd-bb08-42da-933e-3e332b2eb85a" path="/var/lib/kubelet/pods/4327b4bd-bb08-42da-933e-3e332b2eb85a/volumes"
Oct 31 23:15:20.075477 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:20.075447    4301 generic.go:356] "Generic (PLEG): container finished" podID="9c5d8c65-bbe8-486d-b24c-8f6a188f52c3" containerID="0e2af783791d01bacd341053ea1d7073b473c973aa7fcc7f1a971038ceea0881" exitCode=60
Oct 31 23:15:20.075959 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:20.075560    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg" event={"ID":"9c5d8c65-bbe8-486d-b24c-8f6a188f52c3","Type":"ContainerDied","Data":"0e2af783791d01bacd341053ea1d7073b473c973aa7fcc7f1a971038ceea0881"}
Oct 31 23:15:20.075959 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:20.075710    4301 scope.go:122] "RemoveContainer" containerID="8119859baebf4034b1a2ca8b9104c18133bf695404b8c78929c3eae310b97dbf"
Oct 31 23:15:20.076441 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:20.076348    4301 scope.go:122] "RemoveContainer" containerID="0e2af783791d01bacd341053ea1d7073b473c973aa7fcc7f1a971038ceea0881"
Oct 31 23:15:20.076770 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:15:20.076749    4301 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg_odh-ai-gateway-infra(9c5d8c65-bbe8-486d-b24c-8f6a188f52c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg" podUID="9c5d8c65-bbe8-486d-b24c-8f6a188f52c3"
Oct 31 23:15:20.113852 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:20.113814    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg"]
Oct 31 23:15:20.242024 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:20.241996    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8"
Oct 31 23:15:20.307469 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:20.306204    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce874bef-ee28-46ca-aacc-bb04f21486ea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce874bef-ee28-46ca-aacc-bb04f21486ea-maas-api-service-ca\") pod \"ce874bef-ee28-46ca-aacc-bb04f21486ea\" (UID: \"ce874bef-ee28-46ca-aacc-bb04f21486ea\") "
Oct 31 23:15:20.307469 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:20.306476    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce874bef-ee28-46ca-aacc-bb04f21486ea-kube-api-access-x4xjh\" (UniqueName: \"kubernetes.io/projected/ce874bef-ee28-46ca-aacc-bb04f21486ea-kube-api-access-x4xjh\") pod \"ce874bef-ee28-46ca-aacc-bb04f21486ea\" (UID: \"ce874bef-ee28-46ca-aacc-bb04f21486ea\") "
Oct 31 23:15:20.419383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:20.401709    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce874bef-ee28-46ca-aacc-bb04f21486ea-maas-api-service-ca" pod "ce874bef-ee28-46ca-aacc-bb04f21486ea" (UID: "ce874bef-ee28-46ca-aacc-bb04f21486ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:15:20.419383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:20.405535    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce874bef-ee28-46ca-aacc-bb04f21486ea-kube-api-access-x4xjh" pod "ce874bef-ee28-46ca-aacc-bb04f21486ea" (UID: "ce874bef-ee28-46ca-aacc-bb04f21486ea"). InnerVolumeSpecName "kube-api-access-x4xjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:15:20.419383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:20.413659    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce874bef-ee28-46ca-aacc-bb04f21486ea-maas-api-service-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:15:20.419383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:20.413694    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4xjh\" (UniqueName: \"kubernetes.io/projected/ce874bef-ee28-46ca-aacc-bb04f21486ea-kube-api-access-x4xjh\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:15:21.090254 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:21.090210    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8" event={"ID":"ce874bef-ee28-46ca-aacc-bb04f21486ea","Type":"ContainerDied","Data":"9d156cc3183005001b3ae0f8cc80120126f1c09e2e0c205474c20a2f035a4368"}
Oct 31 23:15:21.090767 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:21.090274    4301 scope.go:122] "RemoveContainer" containerID="8d35956df007b8a3010b6b55a5c07b6ec56cd9c4d4a604e88c07814cc6448ca6"
Oct 31 23:15:21.090767 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:21.090216    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8"
Oct 31 23:15:21.597380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:21.591305    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8"]
Oct 31 23:15:21.597380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:21.591431    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-135248-29891475-br7g8"]
Oct 31 23:15:21.784877 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:21.784798    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg"
Oct 31 23:15:21.796377 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:21.796328    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c5d8c65-bbe8-486d-b24c-8f6a188f52c3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c5d8c65-bbe8-486d-b24c-8f6a188f52c3-maas-api-service-ca\") pod \"9c5d8c65-bbe8-486d-b24c-8f6a188f52c3\" (UID: \"9c5d8c65-bbe8-486d-b24c-8f6a188f52c3\") "
Oct 31 23:15:21.796531 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:21.796403    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c5d8c65-bbe8-486d-b24c-8f6a188f52c3-kube-api-access-677kj\" (UniqueName: \"kubernetes.io/projected/9c5d8c65-bbe8-486d-b24c-8f6a188f52c3-kube-api-access-677kj\") pod \"9c5d8c65-bbe8-486d-b24c-8f6a188f52c3\" (UID: \"9c5d8c65-bbe8-486d-b24c-8f6a188f52c3\") "
Oct 31 23:15:21.796778 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:21.796755    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c5d8c65-bbe8-486d-b24c-8f6a188f52c3-maas-api-service-ca" pod "9c5d8c65-bbe8-486d-b24c-8f6a188f52c3" (UID: "9c5d8c65-bbe8-486d-b24c-8f6a188f52c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:15:21.797105 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:21.797082    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c5d8c65-bbe8-486d-b24c-8f6a188f52c3-maas-api-service-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:15:21.798627 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:21.798606    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c5d8c65-bbe8-486d-b24c-8f6a188f52c3-kube-api-access-677kj" pod "9c5d8c65-bbe8-486d-b24c-8f6a188f52c3" (UID: "9c5d8c65-bbe8-486d-b24c-8f6a188f52c3"). InnerVolumeSpecName "kube-api-access-677kj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:15:21.898126 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:21.898093    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-677kj\" (UniqueName: \"kubernetes.io/projected/9c5d8c65-bbe8-486d-b24c-8f6a188f52c3-kube-api-access-677kj\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:15:22.017039 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:22.016976    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac5ff20f-6567b4b69f-sgrvt" podUID="6eb38f55-705c-4b68-a700-9caa0396fc55" containerName="payload-pre-processing" containerID="cri-o://3dbf47a170c027a2c94fdb4176a6706d4a5a1f3c25c97a784936357026ed438c" gracePeriod=25
Oct 31 23:15:22.035150 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:22.035094    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ac5ff20f-6fb5d9b-f9qlx" podUID="cb917671-f386-4be1-99ca-8d7917d484cf" containerName="payload-processing" containerID="cri-o://59d1df9cd0efae0f40bb426b078a5853051fa62cf5834c7401387bcd524543ad" gracePeriod=25
Oct 31 23:15:22.102639 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:22.102615    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg"
Oct 31 23:15:22.103060 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:22.102636    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg" event={"ID":"9c5d8c65-bbe8-486d-b24c-8f6a188f52c3","Type":"ContainerDied","Data":"45127ad2de8f00bdd7c8e2fa8dba43e38426a3b9de696fd2ff8657108f1a0f5f"}
Oct 31 23:15:22.103060 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:22.102715    4301 scope.go:122] "RemoveContainer" containerID="0e2af783791d01bacd341053ea1d7073b473c973aa7fcc7f1a971038ceea0881"
Oct 31 23:15:22.172161 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:22.172120    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg"]
Oct 31 23:15:22.183859 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:22.183825    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7d7eed-29891475-jttsg"]
Oct 31 23:15:22.340847 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:22.340808    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ac5ff20f-6fb5d9b-f9qlx"
Oct 31 23:15:22.393042 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:22.393010    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac5ff20f-6567b4b69f-sgrvt"
Oct 31 23:15:22.405717 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:22.405673    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb917671-f386-4be1-99ca-8d7917d484cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb917671-f386-4be1-99ca-8d7917d484cf-plugins-config-volume\") pod \"cb917671-f386-4be1-99ca-8d7917d484cf\" (UID: \"cb917671-f386-4be1-99ca-8d7917d484cf\") "
Oct 31 23:15:22.405890 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:22.405735    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eb38f55-705c-4b68-a700-9caa0396fc55-kube-api-access-9kcv5\" (UniqueName: \"kubernetes.io/projected/6eb38f55-705c-4b68-a700-9caa0396fc55-kube-api-access-9kcv5\") pod \"6eb38f55-705c-4b68-a700-9caa0396fc55\" (UID: \"6eb38f55-705c-4b68-a700-9caa0396fc55\") "
Oct 31 23:15:22.405890 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:22.405832    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb917671-f386-4be1-99ca-8d7917d484cf-kube-api-access-w2m9n\" (UniqueName: \"kubernetes.io/projected/cb917671-f386-4be1-99ca-8d7917d484cf-kube-api-access-w2m9n\") pod \"cb917671-f386-4be1-99ca-8d7917d484cf\" (UID: \"cb917671-f386-4be1-99ca-8d7917d484cf\") "
Oct 31 23:15:22.406016 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:22.405895    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eb38f55-705c-4b68-a700-9caa0396fc55-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eb38f55-705c-4b68-a700-9caa0396fc55-plugins-config-volume\") pod \"6eb38f55-705c-4b68-a700-9caa0396fc55\" (UID: \"6eb38f55-705c-4b68-a700-9caa0396fc55\") "
Oct 31 23:15:22.406090 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:22.406063    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb917671-f386-4be1-99ca-8d7917d484cf-plugins-config-volume" pod "cb917671-f386-4be1-99ca-8d7917d484cf" (UID: "cb917671-f386-4be1-99ca-8d7917d484cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:15:22.406515 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:22.406484    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eb38f55-705c-4b68-a700-9caa0396fc55-plugins-config-volume" pod "6eb38f55-705c-4b68-a700-9caa0396fc55" (UID: "6eb38f55-705c-4b68-a700-9caa0396fc55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:15:22.406902 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:22.406879    4301 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb917671-f386-4be1-99ca-8d7917d484cf-plugins-config-volume\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:15:22.407009 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:22.406909    4301 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eb38f55-705c-4b68-a700-9caa0396fc55-plugins-config-volume\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:15:22.408161 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:22.408140    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eb38f55-705c-4b68-a700-9caa0396fc55-kube-api-access-9kcv5" pod "6eb38f55-705c-4b68-a700-9caa0396fc55" (UID: "6eb38f55-705c-4b68-a700-9caa0396fc55"). InnerVolumeSpecName "kube-api-access-9kcv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:15:22.408429 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:22.408340    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb917671-f386-4be1-99ca-8d7917d484cf-kube-api-access-w2m9n" pod "cb917671-f386-4be1-99ca-8d7917d484cf" (UID: "cb917671-f386-4be1-99ca-8d7917d484cf"). InnerVolumeSpecName "kube-api-access-w2m9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:15:22.508834 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:22.508805    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kcv5\" (UniqueName: \"kubernetes.io/projected/6eb38f55-705c-4b68-a700-9caa0396fc55-kube-api-access-9kcv5\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:15:22.508834 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:22.508831    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2m9n\" (UniqueName: \"kubernetes.io/projected/cb917671-f386-4be1-99ca-8d7917d484cf-kube-api-access-w2m9n\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:15:23.201400 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.200921    4301 generic.go:356] "Generic (PLEG): container finished" podID="6eb38f55-705c-4b68-a700-9caa0396fc55" containerID="3dbf47a170c027a2c94fdb4176a6706d4a5a1f3c25c97a784936357026ed438c" exitCode=0
Oct 31 23:15:23.201400 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.201013    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac5ff20f-6567b4b69f-sgrvt" event={"ID":"6eb38f55-705c-4b68-a700-9caa0396fc55","Type":"ContainerDied","Data":"3dbf47a170c027a2c94fdb4176a6706d4a5a1f3c25c97a784936357026ed438c"}
Oct 31 23:15:23.201400 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.201057    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac5ff20f-6567b4b69f-sgrvt" event={"ID":"6eb38f55-705c-4b68-a700-9caa0396fc55","Type":"ContainerDied","Data":"1b171859504eb3a925e0203d67cdb5669aa85e364096391dccefdf76185f45f7"}
Oct 31 23:15:23.201400 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.201106    4301 scope.go:122] "RemoveContainer" containerID="3dbf47a170c027a2c94fdb4176a6706d4a5a1f3c25c97a784936357026ed438c"
Oct 31 23:15:23.201400 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.201202    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac5ff20f-6567b4b69f-sgrvt"
Oct 31 23:15:23.204343 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.204315    4301 generic.go:356] "Generic (PLEG): container finished" podID="cb917671-f386-4be1-99ca-8d7917d484cf" containerID="59d1df9cd0efae0f40bb426b078a5853051fa62cf5834c7401387bcd524543ad" exitCode=0
Oct 31 23:15:23.204566 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.204391    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ac5ff20f-6fb5d9b-f9qlx"
Oct 31 23:15:23.204653 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.204410    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ac5ff20f-6fb5d9b-f9qlx" event={"ID":"cb917671-f386-4be1-99ca-8d7917d484cf","Type":"ContainerDied","Data":"59d1df9cd0efae0f40bb426b078a5853051fa62cf5834c7401387bcd524543ad"}
Oct 31 23:15:23.204703 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.204680    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ac5ff20f-6fb5d9b-f9qlx" event={"ID":"cb917671-f386-4be1-99ca-8d7917d484cf","Type":"ContainerDied","Data":"77801ee97e40580c5184a9e3c1ca726319001de0cbf8c7b9f3d72e1e83f9799b"}
Oct 31 23:15:23.215842 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.215823    4301 scope.go:122] "RemoveContainer" containerID="3dbf47a170c027a2c94fdb4176a6706d4a5a1f3c25c97a784936357026ed438c"
Oct 31 23:15:23.216120 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:15:23.216094    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dbf47a170c027a2c94fdb4176a6706d4a5a1f3c25c97a784936357026ed438c\": container with ID starting with 3dbf47a170c027a2c94fdb4176a6706d4a5a1f3c25c97a784936357026ed438c not found: ID does not exist" containerID="3dbf47a170c027a2c94fdb4176a6706d4a5a1f3c25c97a784936357026ed438c"
Oct 31 23:15:23.216197 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.216130    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dbf47a170c027a2c94fdb4176a6706d4a5a1f3c25c97a784936357026ed438c"} err="failed to get container status \"3dbf47a170c027a2c94fdb4176a6706d4a5a1f3c25c97a784936357026ed438c\": rpc error: code = NotFound desc = could not find container \"3dbf47a170c027a2c94fdb4176a6706d4a5a1f3c25c97a784936357026ed438c\": container with ID starting with 3dbf47a170c027a2c94fdb4176a6706d4a5a1f3c25c97a784936357026ed438c not found: ID does not exist"
Oct 31 23:15:23.216197 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.216150    4301 scope.go:122] "RemoveContainer" containerID="59d1df9cd0efae0f40bb426b078a5853051fa62cf5834c7401387bcd524543ad"
Oct 31 23:15:23.227238 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.227219    4301 scope.go:122] "RemoveContainer" containerID="59d1df9cd0efae0f40bb426b078a5853051fa62cf5834c7401387bcd524543ad"
Oct 31 23:15:23.227581 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:15:23.227561    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59d1df9cd0efae0f40bb426b078a5853051fa62cf5834c7401387bcd524543ad\": container with ID starting with 59d1df9cd0efae0f40bb426b078a5853051fa62cf5834c7401387bcd524543ad not found: ID does not exist" containerID="59d1df9cd0efae0f40bb426b078a5853051fa62cf5834c7401387bcd524543ad"
Oct 31 23:15:23.227659 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.227584    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59d1df9cd0efae0f40bb426b078a5853051fa62cf5834c7401387bcd524543ad"} err="failed to get container status \"59d1df9cd0efae0f40bb426b078a5853051fa62cf5834c7401387bcd524543ad\": rpc error: code = NotFound desc = could not find container \"59d1df9cd0efae0f40bb426b078a5853051fa62cf5834c7401387bcd524543ad\": container with ID starting with 59d1df9cd0efae0f40bb426b078a5853051fa62cf5834c7401387bcd524543ad not found: ID does not exist"
Oct 31 23:15:23.228983 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.228963    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ac5ff20f-6567b4b69f-sgrvt"]
Oct 31 23:15:23.241201 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.241166    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ac5ff20f-6567b4b69f-sgrvt"]
Oct 31 23:15:23.252873 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.252849    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ac5ff20f-6fb5d9b-f9qlx"]
Oct 31 23:15:23.260193 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.260166    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ac5ff20f-6fb5d9b-f9qlx"]
Oct 31 23:15:23.404630 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.404597    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eb38f55-705c-4b68-a700-9caa0396fc55" path="/var/lib/kubelet/pods/6eb38f55-705c-4b68-a700-9caa0396fc55/volumes"
Oct 31 23:15:23.405110 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.405094    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c5d8c65-bbe8-486d-b24c-8f6a188f52c3" path="/var/lib/kubelet/pods/9c5d8c65-bbe8-486d-b24c-8f6a188f52c3/volumes"
Oct 31 23:15:23.405599 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.405583    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb917671-f386-4be1-99ca-8d7917d484cf" path="/var/lib/kubelet/pods/cb917671-f386-4be1-99ca-8d7917d484cf/volumes"
Oct 31 23:15:23.406292 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.406279    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce874bef-ee28-46ca-aacc-bb04f21486ea" path="/var/lib/kubelet/pods/ce874bef-ee28-46ca-aacc-bb04f21486ea/volumes"
Oct 31 23:15:23.895609 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.895575    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-907d182f-75bd7d6d7f-6vzjm"]
Oct 31 23:15:23.896177 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896160    4301 cpu_manager.go:414] "containerMap: removing container" podUID="ce874bef-ee28-46ca-aacc-bb04f21486ea" containerName="cleanup"
Oct 31 23:15:23.896303 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896179    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce874bef-ee28-46ca-aacc-bb04f21486ea" containerName="cleanup"
Oct 31 23:15:23.896303 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896195    4301 cpu_manager.go:414] "containerMap: removing container" podUID="9c5d8c65-bbe8-486d-b24c-8f6a188f52c3" containerName="cleanup"
Oct 31 23:15:23.896303 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896202    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c5d8c65-bbe8-486d-b24c-8f6a188f52c3" containerName="cleanup"
Oct 31 23:15:23.896303 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896221    4301 cpu_manager.go:414] "containerMap: removing container" podUID="865bc4e8-2471-4a65-b11e-5ebc94b58fee" containerName="cleanup"
Oct 31 23:15:23.896303 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896228    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="865bc4e8-2471-4a65-b11e-5ebc94b58fee" containerName="cleanup"
Oct 31 23:15:23.896303 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896239    4301 cpu_manager.go:414] "containerMap: removing container" podUID="ce874bef-ee28-46ca-aacc-bb04f21486ea" containerName="cleanup"
Oct 31 23:15:23.896303 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896246    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce874bef-ee28-46ca-aacc-bb04f21486ea" containerName="cleanup"
Oct 31 23:15:23.896303 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896271    4301 cpu_manager.go:414] "containerMap: removing container" podUID="865bc4e8-2471-4a65-b11e-5ebc94b58fee" containerName="cleanup"
Oct 31 23:15:23.896303 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896278    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="865bc4e8-2471-4a65-b11e-5ebc94b58fee" containerName="cleanup"
Oct 31 23:15:23.896303 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896294    4301 cpu_manager.go:414] "containerMap: removing container" podUID="4327b4bd-bb08-42da-933e-3e332b2eb85a" containerName="payload-processing"
Oct 31 23:15:23.896303 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896300    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4327b4bd-bb08-42da-933e-3e332b2eb85a" containerName="payload-processing"
Oct 31 23:15:23.896868 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896320    4301 cpu_manager.go:414] "containerMap: removing container" podUID="3f037628-d51c-48e0-be4e-5d9c99a10332" containerName="istio-proxy"
Oct 31 23:15:23.896868 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896329    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f037628-d51c-48e0-be4e-5d9c99a10332" containerName="istio-proxy"
Oct 31 23:15:23.896868 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896337    4301 cpu_manager.go:414] "containerMap: removing container" podUID="6eb38f55-705c-4b68-a700-9caa0396fc55" containerName="payload-pre-processing"
Oct 31 23:15:23.896868 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896344    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eb38f55-705c-4b68-a700-9caa0396fc55" containerName="payload-pre-processing"
Oct 31 23:15:23.896868 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896385    4301 cpu_manager.go:414] "containerMap: removing container" podUID="cb917671-f386-4be1-99ca-8d7917d484cf" containerName="payload-processing"
Oct 31 23:15:23.896868 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896392    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb917671-f386-4be1-99ca-8d7917d484cf" containerName="payload-processing"
Oct 31 23:15:23.896868 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896436    4301 cpu_manager.go:414] "containerMap: removing container" podUID="09828acd-1f45-4f0b-b2df-dce7f38c2e3d" containerName="cleanup"
Oct 31 23:15:23.896868 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896443    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09828acd-1f45-4f0b-b2df-dce7f38c2e3d" containerName="cleanup"
Oct 31 23:15:23.896868 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896453    4301 cpu_manager.go:414] "containerMap: removing container" podUID="09828acd-1f45-4f0b-b2df-dce7f38c2e3d" containerName="cleanup"
Oct 31 23:15:23.896868 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896460    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09828acd-1f45-4f0b-b2df-dce7f38c2e3d" containerName="cleanup"
Oct 31 23:15:23.896868 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896468    4301 cpu_manager.go:414] "containerMap: removing container" podUID="ce874bef-ee28-46ca-aacc-bb04f21486ea" containerName="cleanup"
Oct 31 23:15:23.896868 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896474    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce874bef-ee28-46ca-aacc-bb04f21486ea" containerName="cleanup"
Oct 31 23:15:23.896868 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896486    4301 cpu_manager.go:414] "containerMap: removing container" podUID="9c5d8c65-bbe8-486d-b24c-8f6a188f52c3" containerName="cleanup"
Oct 31 23:15:23.896868 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896493    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c5d8c65-bbe8-486d-b24c-8f6a188f52c3" containerName="cleanup"
Oct 31 23:15:23.896868 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896640    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="865bc4e8-2471-4a65-b11e-5ebc94b58fee" containerName="cleanup"
Oct 31 23:15:23.896868 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896652    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="4327b4bd-bb08-42da-933e-3e332b2eb85a" containerName="payload-processing"
Oct 31 23:15:23.896868 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896667    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f037628-d51c-48e0-be4e-5d9c99a10332" containerName="istio-proxy"
Oct 31 23:15:23.896868 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896676    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eb38f55-705c-4b68-a700-9caa0396fc55" containerName="payload-pre-processing"
Oct 31 23:15:23.896868 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896685    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c5d8c65-bbe8-486d-b24c-8f6a188f52c3" containerName="cleanup"
Oct 31 23:15:23.896868 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896694    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb917671-f386-4be1-99ca-8d7917d484cf" containerName="payload-processing"
Oct 31 23:15:23.896868 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896703    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="09828acd-1f45-4f0b-b2df-dce7f38c2e3d" containerName="cleanup"
Oct 31 23:15:23.896868 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896716    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="09828acd-1f45-4f0b-b2df-dce7f38c2e3d" containerName="cleanup"
Oct 31 23:15:23.896868 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896725    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce874bef-ee28-46ca-aacc-bb04f21486ea" containerName="cleanup"
Oct 31 23:15:23.896868 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896734    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce874bef-ee28-46ca-aacc-bb04f21486ea" containerName="cleanup"
Oct 31 23:15:23.896868 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896743    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce874bef-ee28-46ca-aacc-bb04f21486ea" containerName="cleanup"
Oct 31 23:15:23.896868 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.896751    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c5d8c65-bbe8-486d-b24c-8f6a188f52c3" containerName="cleanup"
Oct 31 23:15:23.901987 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.901964    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-907d182f-75bd7d6d7f-6vzjm"
Oct 31 23:15:23.913017 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.912930    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-907d182f-75bd7d6d7f-6vzjm"]
Oct 31 23:15:23.935992 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.935962    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/711768db-4ef3-45ec-bea3-8805e1f32d7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-907d182f-75bd7d6d7f-6vzjm\" (UID: \"711768db-4ef3-45ec-bea3-8805e1f32d7c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-907d182f-75bd7d6d7f-6vzjm"
Oct 31 23:15:23.936178 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:23.936167    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58ltj\" (UniqueName: \"kubernetes.io/projected/711768db-4ef3-45ec-bea3-8805e1f32d7c-kube-api-access-58ltj\") pod \"payload-pre-processing-e2e-ait-907d182f-75bd7d6d7f-6vzjm\" (UID: \"711768db-4ef3-45ec-bea3-8805e1f32d7c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-907d182f-75bd7d6d7f-6vzjm"
Oct 31 23:15:24.020465 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:24.014705    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-907d182f-dockercfg-zs9km\""
Oct 31 23:15:24.020465 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:24.014834    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-907d182f\""
Oct 31 23:15:24.086846 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:24.072488    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58ltj\" (UniqueName: \"kubernetes.io/projected/711768db-4ef3-45ec-bea3-8805e1f32d7c-kube-api-access-58ltj\") pod \"payload-pre-processing-e2e-ait-907d182f-75bd7d6d7f-6vzjm\" (UID: \"711768db-4ef3-45ec-bea3-8805e1f32d7c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-907d182f-75bd7d6d7f-6vzjm"
Oct 31 23:15:24.086846 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:24.072753    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/711768db-4ef3-45ec-bea3-8805e1f32d7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-907d182f-75bd7d6d7f-6vzjm\" (UID: \"711768db-4ef3-45ec-bea3-8805e1f32d7c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-907d182f-75bd7d6d7f-6vzjm"
Oct 31 23:15:24.173311 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:24.173234    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/711768db-4ef3-45ec-bea3-8805e1f32d7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-907d182f-75bd7d6d7f-6vzjm\" (UID: \"711768db-4ef3-45ec-bea3-8805e1f32d7c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-907d182f-75bd7d6d7f-6vzjm"
Oct 31 23:15:24.279492 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:24.235193    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58ltj\" (UniqueName: \"kubernetes.io/projected/711768db-4ef3-45ec-bea3-8805e1f32d7c-kube-api-access-58ltj\") pod \"payload-pre-processing-e2e-ait-907d182f-75bd7d6d7f-6vzjm\" (UID: \"711768db-4ef3-45ec-bea3-8805e1f32d7c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-907d182f-75bd7d6d7f-6vzjm"
Oct 31 23:15:24.522978 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:24.522900    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-907d182f-75bd7d6d7f-6vzjm"
Oct 31 23:15:24.861432 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:24.848924    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-907d182f-75bd7d6d7f-6vzjm"]
Oct 31 23:15:25.331664 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:25.331630    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-907d182f-75bd7d6d7f-6vzjm" event={"ID":"711768db-4ef3-45ec-bea3-8805e1f32d7c","Type":"ContainerStarted","Data":"87ed7421699c740444cda7935b9ed25f3e8de513b5bd7993b1288fdcaa445850"}
Oct 31 23:15:25.332058 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:25.331681    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-907d182f-75bd7d6d7f-6vzjm" event={"ID":"711768db-4ef3-45ec-bea3-8805e1f32d7c","Type":"ContainerStarted","Data":"86073d90d8b4845383fa7cb57fa638767baf5e783266b10a3943b57d44e4e071"}
Oct 31 23:15:25.332058 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:25.331769    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-907d182f-75bd7d6d7f-6vzjm"
Oct 31 23:15:25.353434 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:25.353376    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-907d182f-75bd7d6d7f-6vzjm" podStartSLOduration=2.353342582 podStartE2EDuration="2.353342582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:15:25.347788644 +0000 UTC m=+1544.094978156" watchObservedRunningTime="2026-10-31 23:15:25.353342582 +0000 UTC m=+1544.100532092"
Oct 31 23:15:32.548571 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:15:32.548529    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:15:36.341563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:36.341535    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-907d182f-75bd7d6d7f-6vzjm"
Oct 31 23:15:49.845549 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:49.845488    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:49.902558 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:49.902515    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:50.012389 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:50.007706    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 23:15:50.039540 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:50.039491    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:50.198591 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:50.198525    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:50.220431 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:50.219070    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:50.266396 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:50.263792    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:50.330549 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:50.330500    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:50.330920 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:50.330799    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:50.566790 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:50.566694    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded"
Oct 31 23:15:50.854574 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:50.851458    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:50.854574 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:50.853157    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:15:50.971825 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:50.907760    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 23:15:50.982393 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:50.972484    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:15:51.030605 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:51.030559    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 23:15:51.032759 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:51.032714    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:15:51.044714 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:51.044674    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:51.046824 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:51.046781    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:15:51.082945 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:51.082899    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:51.234700 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:51.215580    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:51.234700 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:51.216274    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:15:51.234700 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:51.228567    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:51.244272 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:51.244236    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:15:51.289383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:51.273168    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:51.353413 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:51.306848    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:15:51.354592 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:51.354565    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:15:51.356078 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:51.356059    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:15:51.356665 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:51.338770    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:51.457380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:51.338725    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 23:15:51.459158 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:51.423608    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:15:51.489653 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:51.489528    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"]
Oct 31 23:15:51.615460 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:51.612938    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb" podUID="92fe8257-17fc-4467-9287-b867a88a4f95" containerName="istio-proxy" containerID="cri-o://51afe40fc53888cd8c33e3c5e8968fc3227746e51c71594ef958ce6264ea867c" gracePeriod=30
Oct 31 23:15:51.811246 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:51.709552    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:52.137419 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:52.137383    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 31 23:15:52.174629 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:51.980478    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:15:52.175209 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:52.158730    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:15:52.189415 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:52.187268    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:15:52.189415 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:52.187349    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:15:52.189415 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:52.187415    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:15:52.189415 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:52.187457    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:15:52.189415 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:52.187501    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:15:52.280277 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:52.280248    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:15:52.313230 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:52.313046    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:15:59.129940 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:15:59.129907    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-907d182f-75bd7d6d7f-6vzjm"]
Oct 31 23:16:00.640578 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:00.640541    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:16:00.661952 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:00.661915    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-workload-certs\") pod \"92fe8257-17fc-4467-9287-b867a88a4f95\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") "
Oct 31 23:16:00.662121 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:00.661967    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-istio-envoy\") pod \"92fe8257-17fc-4467-9287-b867a88a4f95\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") "
Oct 31 23:16:00.662121 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:00.662012    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92fe8257-17fc-4467-9287-b867a88a4f95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92fe8257-17fc-4467-9287-b867a88a4f95-istio-podinfo\") pod \"92fe8257-17fc-4467-9287-b867a88a4f95\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") "
Oct 31 23:16:00.662121 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:00.662061    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-credential-socket\") pod \"92fe8257-17fc-4467-9287-b867a88a4f95\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") "
Oct 31 23:16:00.662587 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:00.662283    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-istio-data\") pod \"92fe8257-17fc-4467-9287-b867a88a4f95\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") "
Oct 31 23:16:00.662587 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:00.662376    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92fe8257-17fc-4467-9287-b867a88a4f95-istio-token\" (UniqueName: \"kubernetes.io/projected/92fe8257-17fc-4467-9287-b867a88a4f95-istio-token\") pod \"92fe8257-17fc-4467-9287-b867a88a4f95\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") "
Oct 31 23:16:00.662587 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:00.662409    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92fe8257-17fc-4467-9287-b867a88a4f95-kube-api-access-fzwmx\" (UniqueName: \"kubernetes.io/projected/92fe8257-17fc-4467-9287-b867a88a4f95-kube-api-access-fzwmx\") pod \"92fe8257-17fc-4467-9287-b867a88a4f95\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") "
Oct 31 23:16:00.662587 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:00.662442    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92fe8257-17fc-4467-9287-b867a88a4f95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92fe8257-17fc-4467-9287-b867a88a4f95-istiod-ca-cert\") pod \"92fe8257-17fc-4467-9287-b867a88a4f95\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") "
Oct 31 23:16:00.662587 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:00.662469    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-workload-socket\") pod \"92fe8257-17fc-4467-9287-b867a88a4f95\" (UID: \"92fe8257-17fc-4467-9287-b867a88a4f95\") "
Oct 31 23:16:00.662945 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:00.662716    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-istio-data" pod "92fe8257-17fc-4467-9287-b867a88a4f95" (UID: "92fe8257-17fc-4467-9287-b867a88a4f95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:16:00.663412 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:00.663113    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:16:00.663412 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:00.663335    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-credential-socket" pod "92fe8257-17fc-4467-9287-b867a88a4f95" (UID: "92fe8257-17fc-4467-9287-b867a88a4f95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:16:00.663626 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:00.663563    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-workload-socket" pod "92fe8257-17fc-4467-9287-b867a88a4f95" (UID: "92fe8257-17fc-4467-9287-b867a88a4f95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:16:00.663840 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:00.663810    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-workload-certs" pod "92fe8257-17fc-4467-9287-b867a88a4f95" (UID: "92fe8257-17fc-4467-9287-b867a88a4f95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:16:00.664002 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:00.663982    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92fe8257-17fc-4467-9287-b867a88a4f95-istiod-ca-cert" pod "92fe8257-17fc-4467-9287-b867a88a4f95" (UID: "92fe8257-17fc-4467-9287-b867a88a4f95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:16:00.665169 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:00.665134    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92fe8257-17fc-4467-9287-b867a88a4f95-istio-podinfo" pod "92fe8257-17fc-4467-9287-b867a88a4f95" (UID: "92fe8257-17fc-4467-9287-b867a88a4f95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:16:00.665884 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:00.665774    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-istio-envoy" pod "92fe8257-17fc-4467-9287-b867a88a4f95" (UID: "92fe8257-17fc-4467-9287-b867a88a4f95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:16:00.665884 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:00.665838    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92fe8257-17fc-4467-9287-b867a88a4f95-istio-token" pod "92fe8257-17fc-4467-9287-b867a88a4f95" (UID: "92fe8257-17fc-4467-9287-b867a88a4f95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:16:00.666076 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:00.666055    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92fe8257-17fc-4467-9287-b867a88a4f95-kube-api-access-fzwmx" pod "92fe8257-17fc-4467-9287-b867a88a4f95" (UID: "92fe8257-17fc-4467-9287-b867a88a4f95"). InnerVolumeSpecName "kube-api-access-fzwmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:16:00.764706 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:00.764670    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92fe8257-17fc-4467-9287-b867a88a4f95-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:16:00.764706 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:00.764700    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzwmx\" (UniqueName: \"kubernetes.io/projected/92fe8257-17fc-4467-9287-b867a88a4f95-kube-api-access-fzwmx\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:16:00.764706 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:00.764710    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92fe8257-17fc-4467-9287-b867a88a4f95-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:16:00.764951 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:00.764720    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:16:00.764951 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:00.764728    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:16:00.764951 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:00.764736    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:16:00.764951 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:00.764744    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92fe8257-17fc-4467-9287-b867a88a4f95-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:16:00.764951 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:00.764752    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92fe8257-17fc-4467-9287-b867a88a4f95-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:16:01.323379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:01.322944    4301 generic.go:356] "Generic (PLEG): container finished" podID="92fe8257-17fc-4467-9287-b867a88a4f95" containerID="51afe40fc53888cd8c33e3c5e8968fc3227746e51c71594ef958ce6264ea867c" exitCode=0
Oct 31 23:16:01.323379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:01.323190    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb" event={"ID":"92fe8257-17fc-4467-9287-b867a88a4f95","Type":"ContainerDied","Data":"51afe40fc53888cd8c33e3c5e8968fc3227746e51c71594ef958ce6264ea867c"}
Oct 31 23:16:01.323379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:01.323266    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb" event={"ID":"92fe8257-17fc-4467-9287-b867a88a4f95","Type":"ContainerDied","Data":"b632d1a059c8346b2c48a578d20ec4b3c0526c2f94ffca5403c078cc4e30e590"}
Oct 31 23:16:01.323379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:01.323300    4301 scope.go:122] "RemoveContainer" containerID="51afe40fc53888cd8c33e3c5e8968fc3227746e51c71594ef958ce6264ea867c"
Oct 31 23:16:01.323750 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:01.323697    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"
Oct 31 23:16:01.448391 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:01.440956    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"]
Oct 31 23:16:01.448391 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:01.446112    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-56b15fa9-openshift-default-55894ff4bf-smdlb"]
Oct 31 23:16:01.459148 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:01.458136    4301 scope.go:122] "RemoveContainer" containerID="51afe40fc53888cd8c33e3c5e8968fc3227746e51c71594ef958ce6264ea867c"
Oct 31 23:16:01.459148 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:16:01.458579    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51afe40fc53888cd8c33e3c5e8968fc3227746e51c71594ef958ce6264ea867c\": container with ID starting with 51afe40fc53888cd8c33e3c5e8968fc3227746e51c71594ef958ce6264ea867c not found: ID does not exist" containerID="51afe40fc53888cd8c33e3c5e8968fc3227746e51c71594ef958ce6264ea867c"
Oct 31 23:16:01.459148 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:01.458632    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51afe40fc53888cd8c33e3c5e8968fc3227746e51c71594ef958ce6264ea867c"} err="failed to get container status \"51afe40fc53888cd8c33e3c5e8968fc3227746e51c71594ef958ce6264ea867c\": rpc error: code = NotFound desc = could not find container \"51afe40fc53888cd8c33e3c5e8968fc3227746e51c71594ef958ce6264ea867c\": container with ID starting with 51afe40fc53888cd8c33e3c5e8968fc3227746e51c71594ef958ce6264ea867c not found: ID does not exist"
Oct 31 23:16:02.037641 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:02.037595    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65138e65-4x275"]
Oct 31 23:16:02.044098 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:02.044073    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65138e65-4x275"]
Oct 31 23:16:02.547554 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:16:02.547509    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:16:03.443391 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:03.441915    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92fe8257-17fc-4467-9287-b867a88a4f95" path="/var/lib/kubelet/pods/92fe8257-17fc-4467-9287-b867a88a4f95/volumes"
Oct 31 23:16:03.443391 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:03.442794    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbcf409c-242f-4f2c-ad59-c320f6e27af0" path="/var/lib/kubelet/pods/cbcf409c-242f-4f2c-ad59-c320f6e27af0/volumes"
Oct 31 23:16:04.130599 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.130540    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-907d182f-75bd7d6d7f-6vzjm" podUID="711768db-4ef3-45ec-bea3-8805e1f32d7c" containerName="payload-pre-processing" containerID="cri-o://87ed7421699c740444cda7935b9ed25f3e8de513b5bd7993b1288fdcaa445850" gracePeriod=25
Oct 31 23:16:04.368574 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.368533    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"]
Oct 31 23:16:04.369424 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.369397    4301 cpu_manager.go:414] "containerMap: removing container" podUID="9c5d8c65-bbe8-486d-b24c-8f6a188f52c3" containerName="cleanup"
Oct 31 23:16:04.369608 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.369592    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c5d8c65-bbe8-486d-b24c-8f6a188f52c3" containerName="cleanup"
Oct 31 23:16:04.369752 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.369741    4301 cpu_manager.go:414] "containerMap: removing container" podUID="92fe8257-17fc-4467-9287-b867a88a4f95" containerName="istio-proxy"
Oct 31 23:16:04.369828 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.369818    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92fe8257-17fc-4467-9287-b867a88a4f95" containerName="istio-proxy"
Oct 31 23:16:04.370056 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.370042    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="865bc4e8-2471-4a65-b11e-5ebc94b58fee" containerName="cleanup"
Oct 31 23:16:04.370156 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.370146    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="92fe8257-17fc-4467-9287-b867a88a4f95" containerName="istio-proxy"
Oct 31 23:16:04.370241 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.370232    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c5d8c65-bbe8-486d-b24c-8f6a188f52c3" containerName="cleanup"
Oct 31 23:16:04.374309 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.374291    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:04.381573 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.381496    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-f80dfa-openshift-default-dockercfg-d2v74\""
Oct 31 23:16:04.382165 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.382130    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"]
Oct 31 23:16:04.506127 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.506083    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19145c81-9796-4ca8-92b4-8098589f845e-istiod-ca-cert\") pod \"e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") " pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:04.512606 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.512557    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-workload-socket\") pod \"e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") " pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:04.520918 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.520887    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-istio-data\") pod \"e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") " pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:04.521235 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.521216    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19145c81-9796-4ca8-92b4-8098589f845e-istio-podinfo\") pod \"e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") " pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:04.522125 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.522101    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-workload-certs\") pod \"e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") " pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:04.522283 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.522267    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-istio-envoy\") pod \"e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") " pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:04.522427 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.522409    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-credential-socket\") pod \"e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") " pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:04.522549 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.522531    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhd4r\" (UniqueName: \"kubernetes.io/projected/19145c81-9796-4ca8-92b4-8098589f845e-kube-api-access-vhd4r\") pod \"e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") " pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:04.522727 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.522709    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19145c81-9796-4ca8-92b4-8098589f845e-istio-token\") pod \"e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") " pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:04.568196 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.568170    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-907d182f-75bd7d6d7f-6vzjm"
Oct 31 23:16:04.569128 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.569100    4301 generic.go:356] "Generic (PLEG): container finished" podID="711768db-4ef3-45ec-bea3-8805e1f32d7c" containerID="87ed7421699c740444cda7935b9ed25f3e8de513b5bd7993b1288fdcaa445850" exitCode=0
Oct 31 23:16:04.569559 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.569522    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-907d182f-75bd7d6d7f-6vzjm" event={"ID":"711768db-4ef3-45ec-bea3-8805e1f32d7c","Type":"ContainerDied","Data":"87ed7421699c740444cda7935b9ed25f3e8de513b5bd7993b1288fdcaa445850"}
Oct 31 23:16:04.569746 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.569729    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-907d182f-75bd7d6d7f-6vzjm" event={"ID":"711768db-4ef3-45ec-bea3-8805e1f32d7c","Type":"ContainerDied","Data":"86073d90d8b4845383fa7cb57fa638767baf5e783266b10a3943b57d44e4e071"}
Oct 31 23:16:04.569883 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.569870    4301 scope.go:122] "RemoveContainer" containerID="87ed7421699c740444cda7935b9ed25f3e8de513b5bd7993b1288fdcaa445850"
Oct 31 23:16:04.624306 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.624263    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19145c81-9796-4ca8-92b4-8098589f845e-istiod-ca-cert\") pod \"e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") " pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:04.625681 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.625653    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-workload-socket\") pod \"e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") " pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:04.626202 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.626183    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-istio-data\") pod \"e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") " pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:04.626933 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.626915    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19145c81-9796-4ca8-92b4-8098589f845e-istio-podinfo\") pod \"e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") " pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:04.639526 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.639449    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-workload-certs\") pod \"e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") " pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:04.640463 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.640088    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-istio-envoy\") pod \"e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") " pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:04.640662 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.640631    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-credential-socket\") pod \"e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") " pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:04.641063 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.641044    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhd4r\" (UniqueName: \"kubernetes.io/projected/19145c81-9796-4ca8-92b4-8098589f845e-kube-api-access-vhd4r\") pod \"e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") " pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:04.641273 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.641257    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19145c81-9796-4ca8-92b4-8098589f845e-istio-token\") pod \"e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") " pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:04.641591 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.626802    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-istio-data\") pod \"e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") " pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:04.641755 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.626126    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-workload-socket\") pod \"e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") " pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:04.641871 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.640997    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-credential-socket\") pod \"e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") " pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:04.641977 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.625486    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19145c81-9796-4ca8-92b4-8098589f845e-istiod-ca-cert\") pod \"e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") " pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:04.642348 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.640014    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-workload-certs\") pod \"e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") " pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:04.656623 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.656509    4301 scope.go:122] "RemoveContainer" containerID="87ed7421699c740444cda7935b9ed25f3e8de513b5bd7993b1288fdcaa445850"
Oct 31 23:16:04.657886 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:16:04.657774    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87ed7421699c740444cda7935b9ed25f3e8de513b5bd7993b1288fdcaa445850\": container with ID starting with 87ed7421699c740444cda7935b9ed25f3e8de513b5bd7993b1288fdcaa445850 not found: ID does not exist" containerID="87ed7421699c740444cda7935b9ed25f3e8de513b5bd7993b1288fdcaa445850"
Oct 31 23:16:04.657886 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.657813    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87ed7421699c740444cda7935b9ed25f3e8de513b5bd7993b1288fdcaa445850"} err="failed to get container status \"87ed7421699c740444cda7935b9ed25f3e8de513b5bd7993b1288fdcaa445850\": rpc error: code = NotFound desc = could not find container \"87ed7421699c740444cda7935b9ed25f3e8de513b5bd7993b1288fdcaa445850\": container with ID starting with 87ed7421699c740444cda7935b9ed25f3e8de513b5bd7993b1288fdcaa445850 not found: ID does not exist"
Oct 31 23:16:04.671883 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.671261    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-istio-envoy\") pod \"e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") " pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:04.683451 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.679669    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19145c81-9796-4ca8-92b4-8098589f845e-istio-token\") pod \"e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") " pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:04.709381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.696160    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19145c81-9796-4ca8-92b4-8098589f845e-istio-podinfo\") pod \"e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") " pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:04.709381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.699322    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhd4r\" (UniqueName: \"kubernetes.io/projected/19145c81-9796-4ca8-92b4-8098589f845e-kube-api-access-vhd4r\") pod \"e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") " pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:04.719853 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.718650    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:04.832456 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.830247    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/711768db-4ef3-45ec-bea3-8805e1f32d7c-kube-api-access-58ltj\" (UniqueName: \"kubernetes.io/projected/711768db-4ef3-45ec-bea3-8805e1f32d7c-kube-api-access-58ltj\") pod \"711768db-4ef3-45ec-bea3-8805e1f32d7c\" (UID: \"711768db-4ef3-45ec-bea3-8805e1f32d7c\") "
Oct 31 23:16:04.832456 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.830351    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/711768db-4ef3-45ec-bea3-8805e1f32d7c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/711768db-4ef3-45ec-bea3-8805e1f32d7c-plugins-config-volume\") pod \"711768db-4ef3-45ec-bea3-8805e1f32d7c\" (UID: \"711768db-4ef3-45ec-bea3-8805e1f32d7c\") "
Oct 31 23:16:04.839780 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.839543    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/711768db-4ef3-45ec-bea3-8805e1f32d7c-plugins-config-volume" pod "711768db-4ef3-45ec-bea3-8805e1f32d7c" (UID: "711768db-4ef3-45ec-bea3-8805e1f32d7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:16:04.964512 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:04.951668    4301 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/711768db-4ef3-45ec-bea3-8805e1f32d7c-plugins-config-volume\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:16:05.116068 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:05.116001    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711768db-4ef3-45ec-bea3-8805e1f32d7c-kube-api-access-58ltj" pod "711768db-4ef3-45ec-bea3-8805e1f32d7c" (UID: "711768db-4ef3-45ec-bea3-8805e1f32d7c"). InnerVolumeSpecName "kube-api-access-58ltj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:16:05.165158 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:05.165119    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58ltj\" (UniqueName: \"kubernetes.io/projected/711768db-4ef3-45ec-bea3-8805e1f32d7c-kube-api-access-58ltj\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:16:05.556951 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:05.555322    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"]
Oct 31 23:16:05.561607 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:16:05.561546    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19145c81_9796_4ca8_92b4_8098589f845e.slice/crio-e952bec911559e119429b7c007e972c748ac2c7971b422570180d4cdd1c747e7 WatchSource:0}: Error finding container e952bec911559e119429b7c007e972c748ac2c7971b422570180d4cdd1c747e7: Status 404 returned error can't find the container with id e952bec911559e119429b7c007e972c748ac2c7971b422570180d4cdd1c747e7
Oct 31 23:16:05.566668 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:05.566621    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:16:05.566768 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:05.566720    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:16:05.566823 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:05.566809    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:16:05.579564 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:05.578201    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-907d182f-75bd7d6d7f-6vzjm"
Oct 31 23:16:05.580119 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:05.580092    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6" event={"ID":"19145c81-9796-4ca8-92b4-8098589f845e","Type":"ContainerStarted","Data":"e952bec911559e119429b7c007e972c748ac2c7971b422570180d4cdd1c747e7"}
Oct 31 23:16:05.640776 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:05.640741    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-907d182f-75bd7d6d7f-6vzjm"]
Oct 31 23:16:05.646208 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:05.646180    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-907d182f-75bd7d6d7f-6vzjm"]
Oct 31 23:16:06.603220 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:06.603159    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6" event={"ID":"19145c81-9796-4ca8-92b4-8098589f845e","Type":"ContainerStarted","Data":"3e7307870fa89d74731eb6582dc09ccb516618af4a4ad68d4063f9020d13c385"}
Oct 31 23:16:06.722879 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:06.722845    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:06.783883 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:06.783825    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:06.814033 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:06.813973    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6" podStartSLOduration=2.813959819 podStartE2EDuration="2.813959819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:16:06.651720435 +0000 UTC m=+1585.398909941" watchObservedRunningTime="2026-10-31 23:16:06.813959819 +0000 UTC m=+1585.561149324"
Oct 31 23:16:07.404859 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:07.404823    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="711768db-4ef3-45ec-bea3-8805e1f32d7c" path="/var/lib/kubelet/pods/711768db-4ef3-45ec-bea3-8805e1f32d7c/volumes"
Oct 31 23:16:07.609617 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:07.609584    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:07.610727 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:07.610706    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:10.434019 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:10.433983    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f80dfa-847989bcd7-dcg6p"]
Oct 31 23:16:10.434625 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:10.434610    4301 cpu_manager.go:414] "containerMap: removing container" podUID="711768db-4ef3-45ec-bea3-8805e1f32d7c" containerName="payload-pre-processing"
Oct 31 23:16:10.434688 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:10.434626    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="711768db-4ef3-45ec-bea3-8805e1f32d7c" containerName="payload-pre-processing"
Oct 31 23:16:10.434728 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:10.434709    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="711768db-4ef3-45ec-bea3-8805e1f32d7c" containerName="payload-pre-processing"
Oct 31 23:16:10.438673 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:10.438658    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f80dfa-847989bcd7-dcg6p"
Oct 31 23:16:10.449491 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:10.449466    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-f80dfa\""
Oct 31 23:16:10.451946 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:10.451923    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f80dfa-847989bcd7-dcg6p"]
Oct 31 23:16:10.546843 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:10.546808    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faee4004-8214-4ca7-9dc4-da1a18620c17-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-f80dfa-847989bcd7-dcg6p\" (UID: \"faee4004-8214-4ca7-9dc4-da1a18620c17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f80dfa-847989bcd7-dcg6p"
Oct 31 23:16:10.547007 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:10.546979    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ls9c\" (UniqueName: \"kubernetes.io/projected/faee4004-8214-4ca7-9dc4-da1a18620c17-kube-api-access-5ls9c\") pod \"payload-pre-processing-e2e-shared-a-w1-f80dfa-847989bcd7-dcg6p\" (UID: \"faee4004-8214-4ca7-9dc4-da1a18620c17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f80dfa-847989bcd7-dcg6p"
Oct 31 23:16:10.699430 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:10.698489    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ls9c\" (UniqueName: \"kubernetes.io/projected/faee4004-8214-4ca7-9dc4-da1a18620c17-kube-api-access-5ls9c\") pod \"payload-pre-processing-e2e-shared-a-w1-f80dfa-847989bcd7-dcg6p\" (UID: \"faee4004-8214-4ca7-9dc4-da1a18620c17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f80dfa-847989bcd7-dcg6p"
Oct 31 23:16:10.699430 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:10.698821    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faee4004-8214-4ca7-9dc4-da1a18620c17-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-f80dfa-847989bcd7-dcg6p\" (UID: \"faee4004-8214-4ca7-9dc4-da1a18620c17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f80dfa-847989bcd7-dcg6p"
Oct 31 23:16:10.790085 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:10.790050    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faee4004-8214-4ca7-9dc4-da1a18620c17-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-f80dfa-847989bcd7-dcg6p\" (UID: \"faee4004-8214-4ca7-9dc4-da1a18620c17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f80dfa-847989bcd7-dcg6p"
Oct 31 23:16:10.807896 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:10.807859    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ls9c\" (UniqueName: \"kubernetes.io/projected/faee4004-8214-4ca7-9dc4-da1a18620c17-kube-api-access-5ls9c\") pod \"payload-pre-processing-e2e-shared-a-w1-f80dfa-847989bcd7-dcg6p\" (UID: \"faee4004-8214-4ca7-9dc4-da1a18620c17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f80dfa-847989bcd7-dcg6p"
Oct 31 23:16:11.055323 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:11.055236    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f80dfa-847989bcd7-dcg6p"
Oct 31 23:16:11.340222 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:11.340181    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f80dfa-847989bcd7-dcg6p"]
Oct 31 23:16:11.341519 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:16:11.341493    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaee4004_8214_4ca7_9dc4_da1a18620c17.slice/crio-8aeab6b08e8ac279f280f979cc27fccfc1187dcfb1c6503ec88d0197a0e57ad1 WatchSource:0}: Error finding container 8aeab6b08e8ac279f280f979cc27fccfc1187dcfb1c6503ec88d0197a0e57ad1: Status 404 returned error can't find the container with id 8aeab6b08e8ac279f280f979cc27fccfc1187dcfb1c6503ec88d0197a0e57ad1
Oct 31 23:16:11.764937 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:11.764892    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f80dfa-847989bcd7-dcg6p" event={"ID":"faee4004-8214-4ca7-9dc4-da1a18620c17","Type":"ContainerStarted","Data":"5e73ef3a2e757e8886446d510f728e56d30c75c9780c3bf40494967422e6084b"}
Oct 31 23:16:11.765329 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:11.765071    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f80dfa-847989bcd7-dcg6p"
Oct 31 23:16:11.765329 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:11.765105    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f80dfa-847989bcd7-dcg6p" event={"ID":"faee4004-8214-4ca7-9dc4-da1a18620c17","Type":"ContainerStarted","Data":"8aeab6b08e8ac279f280f979cc27fccfc1187dcfb1c6503ec88d0197a0e57ad1"}
Oct 31 23:16:11.785898 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:11.785849    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f80dfa-847989bcd7-dcg6p" podStartSLOduration=1.785833168 podStartE2EDuration="1.785833168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:16:11.781665321 +0000 UTC m=+1590.528854825" watchObservedRunningTime="2026-10-31 23:16:11.785833168 +0000 UTC m=+1590.533022673"
Oct 31 23:16:22.809622 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:22.809568    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f80dfa-847989bcd7-dcg6p"
Oct 31 23:16:25.909592 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:25.909554    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:16:27.799660 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:27.799625    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d6685e-7c669d9694-8jd86"]
Oct 31 23:16:27.805867 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:27.805840    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6685e-7c669d9694-8jd86"
Oct 31 23:16:27.806210 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:27.806180    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6685e-7bdfff8567-s9nqv"]
Oct 31 23:16:27.815034 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:27.815005    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-d6685e\""
Oct 31 23:16:27.815213 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:27.815118    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-d6685e-dockercfg-975pj\""
Oct 31 23:16:27.817521 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:27.817498    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6685e-7bdfff8567-s9nqv"
Oct 31 23:16:27.819247 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:27.819227    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d6685e-7c669d9694-8jd86"]
Oct 31 23:16:27.847326 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:27.847286    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fn97\" (UniqueName: \"kubernetes.io/projected/b9eac259-b8e2-4fff-a019-2daf3c17c47b-kube-api-access-5fn97\") pod \"payload-processing-e2e-shared-b-w4-d6685e-7c669d9694-8jd86\" (UID: \"b9eac259-b8e2-4fff-a019-2daf3c17c47b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6685e-7c669d9694-8jd86"
Oct 31 23:16:27.861033 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:27.860992    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35ceb2a4-953d-4719-9d85-e3c8491d1825-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-d6685e-7bdfff8567-s9nqv\" (UID: \"35ceb2a4-953d-4719-9d85-e3c8491d1825\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6685e-7bdfff8567-s9nqv"
Oct 31 23:16:27.862571 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:27.862539    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9eac259-b8e2-4fff-a019-2daf3c17c47b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-d6685e-7c669d9694-8jd86\" (UID: \"b9eac259-b8e2-4fff-a019-2daf3c17c47b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6685e-7c669d9694-8jd86"
Oct 31 23:16:27.866692 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:27.866663    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfxng\" (UniqueName: \"kubernetes.io/projected/35ceb2a4-953d-4719-9d85-e3c8491d1825-kube-api-access-qfxng\") pod \"payload-pre-processing-e2e-shared-b-w4-d6685e-7bdfff8567-s9nqv\" (UID: \"35ceb2a4-953d-4719-9d85-e3c8491d1825\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6685e-7bdfff8567-s9nqv"
Oct 31 23:16:27.909786 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:27.909747    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6685e-7bdfff8567-s9nqv"]
Oct 31 23:16:28.012964 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:28.008255    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfxng\" (UniqueName: \"kubernetes.io/projected/35ceb2a4-953d-4719-9d85-e3c8491d1825-kube-api-access-qfxng\") pod \"payload-pre-processing-e2e-shared-b-w4-d6685e-7bdfff8567-s9nqv\" (UID: \"35ceb2a4-953d-4719-9d85-e3c8491d1825\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6685e-7bdfff8567-s9nqv"
Oct 31 23:16:28.012964 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:28.008403    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fn97\" (UniqueName: \"kubernetes.io/projected/b9eac259-b8e2-4fff-a019-2daf3c17c47b-kube-api-access-5fn97\") pod \"payload-processing-e2e-shared-b-w4-d6685e-7c669d9694-8jd86\" (UID: \"b9eac259-b8e2-4fff-a019-2daf3c17c47b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6685e-7c669d9694-8jd86"
Oct 31 23:16:28.012964 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:28.008607    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35ceb2a4-953d-4719-9d85-e3c8491d1825-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-d6685e-7bdfff8567-s9nqv\" (UID: \"35ceb2a4-953d-4719-9d85-e3c8491d1825\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6685e-7bdfff8567-s9nqv"
Oct 31 23:16:28.012964 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:28.008711    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9eac259-b8e2-4fff-a019-2daf3c17c47b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-d6685e-7c669d9694-8jd86\" (UID: \"b9eac259-b8e2-4fff-a019-2daf3c17c47b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6685e-7c669d9694-8jd86"
Oct 31 23:16:28.012964 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:28.009977    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35ceb2a4-953d-4719-9d85-e3c8491d1825-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-d6685e-7bdfff8567-s9nqv\" (UID: \"35ceb2a4-953d-4719-9d85-e3c8491d1825\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6685e-7bdfff8567-s9nqv"
Oct 31 23:16:28.020393 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:28.018638    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9eac259-b8e2-4fff-a019-2daf3c17c47b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-d6685e-7c669d9694-8jd86\" (UID: \"b9eac259-b8e2-4fff-a019-2daf3c17c47b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6685e-7c669d9694-8jd86"
Oct 31 23:16:28.134439 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:28.133069    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfxng\" (UniqueName: \"kubernetes.io/projected/35ceb2a4-953d-4719-9d85-e3c8491d1825-kube-api-access-qfxng\") pod \"payload-pre-processing-e2e-shared-b-w4-d6685e-7bdfff8567-s9nqv\" (UID: \"35ceb2a4-953d-4719-9d85-e3c8491d1825\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6685e-7bdfff8567-s9nqv"
Oct 31 23:16:28.157072 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:28.156820    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6685e-7bdfff8567-s9nqv"
Oct 31 23:16:28.160479 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:28.160030    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fn97\" (UniqueName: \"kubernetes.io/projected/b9eac259-b8e2-4fff-a019-2daf3c17c47b-kube-api-access-5fn97\") pod \"payload-processing-e2e-shared-b-w4-d6685e-7c669d9694-8jd86\" (UID: \"b9eac259-b8e2-4fff-a019-2daf3c17c47b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6685e-7c669d9694-8jd86"
Oct 31 23:16:28.451869 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:28.451781    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6685e-7c669d9694-8jd86"
Oct 31 23:16:28.962190 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:28.962165    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6685e-7bdfff8567-s9nqv"]
Oct 31 23:16:28.963604 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:16:28.963576    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35ceb2a4_953d_4719_9d85_e3c8491d1825.slice/crio-a0127463ea4d8b83946181fbb5b25dbcfce0800e54baa1531929f8765a47c55e WatchSource:0}: Error finding container a0127463ea4d8b83946181fbb5b25dbcfce0800e54baa1531929f8765a47c55e: Status 404 returned error can't find the container with id a0127463ea4d8b83946181fbb5b25dbcfce0800e54baa1531929f8765a47c55e
Oct 31 23:16:28.997286 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:28.997260    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d6685e-7c669d9694-8jd86"]
Oct 31 23:16:29.000620 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:16:29.000592    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9eac259_b8e2_4fff_a019_2daf3c17c47b.slice/crio-969e6f259c88dff26a8a61754ccecc755d2f029743bcf767860064d14f57b8b3 WatchSource:0}: Error finding container 969e6f259c88dff26a8a61754ccecc755d2f029743bcf767860064d14f57b8b3: Status 404 returned error can't find the container with id 969e6f259c88dff26a8a61754ccecc755d2f029743bcf767860064d14f57b8b3
Oct 31 23:16:29.404870 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:29.404834    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6685e-7bdfff8567-s9nqv" event={"ID":"35ceb2a4-953d-4719-9d85-e3c8491d1825","Type":"ContainerStarted","Data":"49034d96749eaaea772cacd57f65025100f34fae047aeb6bfec1a05d795eb028"}
Oct 31 23:16:29.405048 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:29.404894    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6685e-7bdfff8567-s9nqv"
Oct 31 23:16:29.405048 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:29.404917    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6685e-7c669d9694-8jd86"
Oct 31 23:16:29.405048 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:29.404937    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6685e-7bdfff8567-s9nqv" event={"ID":"35ceb2a4-953d-4719-9d85-e3c8491d1825","Type":"ContainerStarted","Data":"a0127463ea4d8b83946181fbb5b25dbcfce0800e54baa1531929f8765a47c55e"}
Oct 31 23:16:29.405048 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:29.404967    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6685e-7c669d9694-8jd86" event={"ID":"b9eac259-b8e2-4fff-a019-2daf3c17c47b","Type":"ContainerStarted","Data":"57d9d967eb5a8fe9fb27f47d3ebf2d1dbd43d04c7d4858ec17807d9dd41e58e0"}
Oct 31 23:16:29.405048 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:29.404993    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6685e-7c669d9694-8jd86" event={"ID":"b9eac259-b8e2-4fff-a019-2daf3c17c47b","Type":"ContainerStarted","Data":"969e6f259c88dff26a8a61754ccecc755d2f029743bcf767860064d14f57b8b3"}
Oct 31 23:16:29.423254 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:29.423204    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6685e-7bdfff8567-s9nqv" podStartSLOduration=2.423191128 podStartE2EDuration="2.423191128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:16:29.421145048 +0000 UTC m=+1608.168334554" watchObservedRunningTime="2026-10-31 23:16:29.423191128 +0000 UTC m=+1608.170380632"
Oct 31 23:16:29.458576 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:29.458511    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6685e-7c669d9694-8jd86" podStartSLOduration=2.458491806 podStartE2EDuration="2.458491806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:16:29.446717833 +0000 UTC m=+1608.193907338" watchObservedRunningTime="2026-10-31 23:16:29.458491806 +0000 UTC m=+1608.205681312"
Oct 31 23:16:32.564608 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:16:32.564564    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:16:35.135435 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:35.135393    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"]
Oct 31 23:16:35.143753 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:35.143720    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:16:35.153035 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:35.153000    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"]
Oct 31 23:16:35.155117 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:35.155091    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-7c4c91-openshift-default-dockercfg-ss775\""
Oct 31 23:16:35.164034 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:35.164003    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-workload-certs\") pod \"e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") " pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:16:35.164173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:35.164141    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istio-data\") pod \"e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") " pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:16:35.164248 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:35.164173    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istiod-ca-cert\") pod \"e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") " pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:16:35.164248 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:35.164223    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istio-envoy\") pod \"e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") " pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:16:35.164248 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:35.164246    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istio-token\") pod \"e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") " pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:16:35.164503 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:35.164288    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-credential-socket\") pod \"e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") " pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:16:35.164503 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:35.164389    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pg97\" (UniqueName: \"kubernetes.io/projected/9a58ab83-2394-48a8-aa4f-04c22ee67a24-kube-api-access-8pg97\") pod \"e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") " pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:16:35.164503 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:35.164431    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istio-podinfo\") pod \"e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") " pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:16:35.164503 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:35.164469    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-workload-socket\") pod \"e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") " pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:16:35.274390 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:35.272616    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istio-envoy\") pod \"e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") " pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:16:35.274390 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:35.272672    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istio-token\") pod \"e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") " pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:16:35.274390 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:35.272730    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-credential-socket\") pod \"e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") " pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:16:35.274390 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:35.272834    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pg97\" (UniqueName: \"kubernetes.io/projected/9a58ab83-2394-48a8-aa4f-04c22ee67a24-kube-api-access-8pg97\") pod \"e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") " pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:16:35.274390 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:35.272866    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istio-podinfo\") pod \"e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") " pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:16:35.274390 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:35.272906    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-workload-socket\") pod \"e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") " pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:16:35.274390 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:35.272983    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-workload-certs\") pod \"e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") " pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:16:35.274390 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:35.273229    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istio-data\") pod \"e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") " pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:16:35.274390 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:35.273276    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istiod-ca-cert\") pod \"e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") " pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:16:35.274390 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:35.274217    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istiod-ca-cert\") pod \"e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") " pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:16:35.284731 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:35.284066    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-credential-socket\") pod \"e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") " pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:16:35.289767 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:35.289738    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-workload-certs\") pod \"e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") " pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:16:35.290251 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:35.290227    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-workload-socket\") pod \"e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") " pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:16:35.290623 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:35.290603    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istio-data\") pod \"e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") " pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:16:35.308221 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:35.305853    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istio-podinfo\") pod \"e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") " pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:16:35.332811 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:35.332728    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pg97\" (UniqueName: \"kubernetes.io/projected/9a58ab83-2394-48a8-aa4f-04c22ee67a24-kube-api-access-8pg97\") pod \"e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") " pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:16:35.340290 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:35.336383    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istio-envoy\") pod \"e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") " pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:16:35.373386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:35.369746    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istio-token\") pod \"e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") " pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:16:35.474726 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:35.474641    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:16:36.168619 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:36.167551    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"]
Oct 31 23:16:36.171728 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:16:36.171693    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a58ab83_2394_48a8_aa4f_04c22ee67a24.slice/crio-2de3b9cb2acb2f6b7d0bacaf1fe3c9cbd480f6f9d1d1b27d0e8f5668b4ca0b1f WatchSource:0}: Error finding container 2de3b9cb2acb2f6b7d0bacaf1fe3c9cbd480f6f9d1d1b27d0e8f5668b4ca0b1f: Status 404 returned error can't find the container with id 2de3b9cb2acb2f6b7d0bacaf1fe3c9cbd480f6f9d1d1b27d0e8f5668b4ca0b1f
Oct 31 23:16:36.174945 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:36.174906    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:16:36.175074 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:36.174974    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:16:36.175074 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:36.175007    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:16:36.558083 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:36.557983    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx" event={"ID":"9a58ab83-2394-48a8-aa4f-04c22ee67a24","Type":"ContainerStarted","Data":"d48357bb04b7e1e03a059677a5ad42bbccdf1017aba1ca95dad92fa740178e30"}
Oct 31 23:16:36.558083 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:36.558053    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx" event={"ID":"9a58ab83-2394-48a8-aa4f-04c22ee67a24","Type":"ContainerStarted","Data":"2de3b9cb2acb2f6b7d0bacaf1fe3c9cbd480f6f9d1d1b27d0e8f5668b4ca0b1f"}
Oct 31 23:16:36.588708 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:36.588646    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx" podStartSLOduration=1.5886268449999998 podStartE2EDuration="1.588626845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:16:36.586454002 +0000 UTC m=+1615.333643508" watchObservedRunningTime="2026-10-31 23:16:36.588626845 +0000 UTC m=+1615.335816351"
Oct 31 23:16:37.475550 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.475515    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:16:37.481337 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.481305    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:16:37.536799 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.536766    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"]
Oct 31 23:16:37.542787 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.542763    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:16:37.549832 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.549801    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-8e6a08-openshift-default-dockercfg-btdj9\""
Oct 31 23:16:37.562131 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.562101    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"]
Oct 31 23:16:37.568709 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.568632    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:16:37.570984 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.570954    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:16:37.640283 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.640241    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istio-envoy\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:16:37.640525 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.640471    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-credential-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:16:37.640602 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.640538    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istio-podinfo\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:16:37.640672 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.640659    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istio-token\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:16:37.640729 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.640685    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:16:37.640846 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.640827    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-workload-certs\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:16:37.640910 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.640872    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzr92\" (UniqueName: \"kubernetes.io/projected/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-kube-api-access-qzr92\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:16:37.640967 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.640921    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-workload-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:16:37.641023 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.640972    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istio-data\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:16:37.748565 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.748469    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-workload-certs\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:16:37.753675 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.753645    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzr92\" (UniqueName: \"kubernetes.io/projected/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-kube-api-access-qzr92\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:16:37.753982 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.753954    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-workload-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:16:37.765612 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.765573    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istio-data\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:16:37.774383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.765887    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istio-envoy\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:16:37.774383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.766075    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-credential-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:16:37.774383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.766266    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istio-podinfo\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:16:37.774383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.766405    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istio-token\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:16:37.774383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.766436    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:16:37.774383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.767636    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:16:37.775209 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.775184    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istio-data\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:16:37.792385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.775669    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-workload-certs\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:16:37.793084 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.777740    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-credential-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:16:37.793257 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.787286    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-workload-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:16:37.825178 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.825134    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzr92\" (UniqueName: \"kubernetes.io/projected/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-kube-api-access-qzr92\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:16:37.826581 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.826523    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istio-envoy\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:16:37.828207 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.828184    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istio-token\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:16:37.830223 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.830176    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istio-podinfo\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:16:37.866468 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:37.866434    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:16:38.155937 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:38.155907    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"]
Oct 31 23:16:38.156491 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:16:38.156464    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d5e2ad8_17f8_412f_b2c2_24b93d77d1a2.slice/crio-6133ed3031ee5d16e93ceb59667becd63f2c36dc8afcc8d138cf93d3f43bd4d7 WatchSource:0}: Error finding container 6133ed3031ee5d16e93ceb59667becd63f2c36dc8afcc8d138cf93d3f43bd4d7: Status 404 returned error can't find the container with id 6133ed3031ee5d16e93ceb59667becd63f2c36dc8afcc8d138cf93d3f43bd4d7
Oct 31 23:16:38.159481 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:38.159439    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:16:38.159559 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:38.159537    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:16:38.159617 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:38.159596    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:16:38.593466 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:38.591999    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6" event={"ID":"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2","Type":"ContainerStarted","Data":"97b51d48183b8e6ee01153c6871cd74b0c86ab1ad7251bb37a7669a07445a75f"}
Oct 31 23:16:38.593466 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:38.592058    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6" event={"ID":"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2","Type":"ContainerStarted","Data":"6133ed3031ee5d16e93ceb59667becd63f2c36dc8afcc8d138cf93d3f43bd4d7"}
Oct 31 23:16:38.617655 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:38.617588    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6" podStartSLOduration=1.617571269 podStartE2EDuration="1.617571269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:16:38.612812682 +0000 UTC m=+1617.360002189" watchObservedRunningTime="2026-10-31 23:16:38.617571269 +0000 UTC m=+1617.364760780"
Oct 31 23:16:38.650577 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:38.650504    4301 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 23:16:38.650756 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:38.650574    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6" podUID="19145c81-9796-4ca8-92b4-8098589f845e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:16:38.867343 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:38.867268    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:16:39.599525 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:39.599446    4301 generic.go:356] "Generic (PLEG): container finished" podID="19145c81-9796-4ca8-92b4-8098589f845e" containerID="3e7307870fa89d74731eb6582dc09ccb516618af4a4ad68d4063f9020d13c385" exitCode=0
Oct 31 23:16:39.599975 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:39.599517    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6" event={"ID":"19145c81-9796-4ca8-92b4-8098589f845e","Type":"ContainerDied","Data":"3e7307870fa89d74731eb6582dc09ccb516618af4a4ad68d4063f9020d13c385"}
Oct 31 23:16:39.599975 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:39.599970    4301 scope.go:122] "RemoveContainer" containerID="3e7307870fa89d74731eb6582dc09ccb516618af4a4ad68d4063f9020d13c385"
Oct 31 23:16:39.601457 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:39.601427    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:16:39.601550 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:39.601493    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:16:39.601550 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:39.601523    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:16:39.882386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:39.882292    4301 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 31 23:16:39.882587 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:39.882393    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6" podUID="8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 31 23:16:40.458427 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:40.455832    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6685e-7c669d9694-8jd86"
Oct 31 23:16:40.772240 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:40.772152    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6" event={"ID":"19145c81-9796-4ca8-92b4-8098589f845e","Type":"ContainerStarted","Data":"b243d9f2ce515e523a8a9a44b6bb247480f65fd932bb9d12dcd9a9cbce1a0879"}
Oct 31 23:16:40.883395 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:40.882620    4301 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 31 23:16:40.883395 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:40.882687    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6" podUID="8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 31 23:16:41.481433 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:41.480340    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6685e-7bdfff8567-s9nqv"
Oct 31 23:16:41.730381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:41.723982    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:41.740480 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:41.734038    4301 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 23:16:41.740480 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:41.734117    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6" podUID="19145c81-9796-4ca8-92b4-8098589f845e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 23:16:41.874444 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:41.874400    4301 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 31 23:16:41.875043 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:41.874996    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6" podUID="8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 31 23:16:42.721371 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:42.721318    4301 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 23:16:42.721702 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:42.721656    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6" podUID="19145c81-9796-4ca8-92b4-8098589f845e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 23:16:42.867816 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:42.867776    4301 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 31 23:16:42.868110 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:42.868084    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6" podUID="8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 31 23:16:43.719850 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:43.719810    4301 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 23:16:43.720480 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:43.720446    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6" podUID="19145c81-9796-4ca8-92b4-8098589f845e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 23:16:43.886880 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:43.886849    4301 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 31 23:16:43.887176 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:43.887143    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6" podUID="8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 31 23:16:44.719652 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:44.719613    4301 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 23:16:44.719838 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:44.719671    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6" podUID="19145c81-9796-4ca8-92b4-8098589f845e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 23:16:44.868388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:44.867767    4301 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 31 23:16:44.868388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:44.867834    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6" podUID="8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 31 23:16:45.723973 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:45.723932    4301 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 23:16:45.724247 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:45.724218    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6" podUID="19145c81-9796-4ca8-92b4-8098589f845e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 23:16:45.872556 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:45.872526    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:16:45.873025 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:45.872828    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:16:45.873632 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:45.873601    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:16:46.719792 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:46.719755    4301 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 23:16:46.719970 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:46.719807    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6" podUID="19145c81-9796-4ca8-92b4-8098589f845e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 23:16:47.724221 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:47.724191    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:47.724678 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:47.724571    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:47.725230 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:47.725210    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:16:49.258113 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:49.258077    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"]
Oct 31 23:16:49.264893 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:49.264869    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:16:49.269243 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:49.269199    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-36a15fc5-openshift-default-dockercfg-886f5\""
Oct 31 23:16:49.273575 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:49.273510    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"]
Oct 31 23:16:49.326955 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:49.326916    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istio-data\") pod \"e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") " pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:16:49.327152 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:49.326983    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istio-podinfo\") pod \"e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") " pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:16:49.327152 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:49.327005    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-credential-socket\") pod \"e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") " pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:16:49.327152 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:49.327150    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-workload-socket\") pod \"e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") " pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:16:49.327339 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:49.327167    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istio-token\") pod \"e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") " pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:16:49.327339 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:49.327187    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istio-envoy\") pod \"e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") " pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:16:49.327339 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:49.327253    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6cfr\" (UniqueName: \"kubernetes.io/projected/a3ef6611-0d56-41c3-9357-81e33a2fd67a-kube-api-access-q6cfr\") pod \"e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") " pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:16:49.327339 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:49.327279    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-workload-certs\") pod \"e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") " pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:16:49.327339 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:49.327298    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istiod-ca-cert\") pod \"e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") " pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:16:49.444580 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:49.441837    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6cfr\" (UniqueName: \"kubernetes.io/projected/a3ef6611-0d56-41c3-9357-81e33a2fd67a-kube-api-access-q6cfr\") pod \"e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") " pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:16:49.453491 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:49.453452    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-workload-certs\") pod \"e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") " pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:16:49.454220 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:49.454195    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istiod-ca-cert\") pod \"e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") " pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:16:49.469415 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:49.454481    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istio-data\") pod \"e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") " pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:16:49.470825 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:49.470800    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istio-podinfo\") pod \"e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") " pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:16:49.471783 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:49.471764    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-credential-socket\") pod \"e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") " pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:16:49.472514 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:49.472489    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-workload-socket\") pod \"e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") " pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:16:49.472964 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:49.472945    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istio-token\") pod \"e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") " pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:16:49.473497 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:49.473476    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istio-envoy\") pod \"e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") " pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:16:49.505506 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:49.472139    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-credential-socket\") pod \"e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") " pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:16:49.505506 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:49.470669    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istiod-ca-cert\") pod \"e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") " pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:16:49.505506 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:49.472890    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-workload-socket\") pod \"e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") " pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:16:49.505506 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:49.470027    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istio-data\") pod \"e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") " pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:16:49.505506 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:49.454135    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-workload-certs\") pod \"e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") " pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:16:49.506216 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:49.506187    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istio-envoy\") pod \"e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") " pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:16:49.507600 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:49.506957    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istio-podinfo\") pod \"e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") " pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:16:49.530728 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:49.530132    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6cfr\" (UniqueName: \"kubernetes.io/projected/a3ef6611-0d56-41c3-9357-81e33a2fd67a-kube-api-access-q6cfr\") pod \"e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") " pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:16:49.533751 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:49.533693    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istio-token\") pod \"e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") " pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:16:49.588978 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:49.588691    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:16:50.452690 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:50.450057    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"]
Oct 31 23:16:50.454305 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:16:50.454276    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ef6611_0d56_41c3_9357_81e33a2fd67a.slice/crio-556cf16412aee045356afec034ce8b88ea9a37307a66cca5f57414a2a2c5bbca WatchSource:0}: Error finding container 556cf16412aee045356afec034ce8b88ea9a37307a66cca5f57414a2a2c5bbca: Status 404 returned error can't find the container with id 556cf16412aee045356afec034ce8b88ea9a37307a66cca5f57414a2a2c5bbca
Oct 31 23:16:50.457182 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:50.457143    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:16:50.457296 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:50.457217    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:16:50.457296 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:50.457248    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:16:51.129505 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:51.129461    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz" event={"ID":"a3ef6611-0d56-41c3-9357-81e33a2fd67a","Type":"ContainerStarted","Data":"cb73ef2b0577265122157fc5171ab5630a7402bac27008f6b293da6d3ba371d3"}
Oct 31 23:16:51.129746 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:51.129565    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz" event={"ID":"a3ef6611-0d56-41c3-9357-81e33a2fd67a","Type":"ContainerStarted","Data":"556cf16412aee045356afec034ce8b88ea9a37307a66cca5f57414a2a2c5bbca"}
Oct 31 23:16:51.184909 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:51.184846    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz" podStartSLOduration=2.184825452 podStartE2EDuration="2.184825452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:16:51.181741652 +0000 UTC m=+1629.928931158" watchObservedRunningTime="2026-10-31 23:16:51.184825452 +0000 UTC m=+1629.932014959"
Oct 31 23:16:51.589906 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:51.589812    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:16:51.595546 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:51.595509    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:16:52.136453 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:52.136423    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:16:52.137343 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:52.137324    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:16:59.826367 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:59.826320    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10903-01497c"]
Oct 31 23:16:59.830586 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:59.830568    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10903-01497c"
Oct 31 23:16:59.838371 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:59.837910    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10903-01497c"]
Oct 31 23:16:59.883194 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:16:59.883163    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-rq76x\""
Oct 31 23:17:00.012969 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:00.012929    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbd4f\" (UniqueName: \"kubernetes.io/projected/34ee2aba-8005-48b9-9cd8-459efcc498c5-kube-api-access-gbd4f\") pod \"test-curl-10903-01497c\" (UID: \"34ee2aba-8005-48b9-9cd8-459efcc498c5\") " pod="opendatahub/test-curl-10903-01497c"
Oct 31 23:17:00.138445 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:00.131255    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbd4f\" (UniqueName: \"kubernetes.io/projected/34ee2aba-8005-48b9-9cd8-459efcc498c5-kube-api-access-gbd4f\") pod \"test-curl-10903-01497c\" (UID: \"34ee2aba-8005-48b9-9cd8-459efcc498c5\") " pod="opendatahub/test-curl-10903-01497c"
Oct 31 23:17:00.174381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:00.173990    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbd4f\" (UniqueName: \"kubernetes.io/projected/34ee2aba-8005-48b9-9cd8-459efcc498c5-kube-api-access-gbd4f\") pod \"test-curl-10903-01497c\" (UID: \"34ee2aba-8005-48b9-9cd8-459efcc498c5\") " pod="opendatahub/test-curl-10903-01497c"
Oct 31 23:17:00.456922 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:00.456847    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10903-01497c"
Oct 31 23:17:00.739850 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:00.739827    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10903-01497c"]
Oct 31 23:17:00.741539 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:17:00.741504    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34ee2aba_8005_48b9_9cd8_459efcc498c5.slice/crio-02817dcd58f8a67eb5f5e7019b858b2925e0cb4dbf69a7c6a395578d4eacd1d9 WatchSource:0}: Error finding container 02817dcd58f8a67eb5f5e7019b858b2925e0cb4dbf69a7c6a395578d4eacd1d9: Status 404 returned error can't find the container with id 02817dcd58f8a67eb5f5e7019b858b2925e0cb4dbf69a7c6a395578d4eacd1d9
Oct 31 23:17:01.261520 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:01.261484    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10903-01497c" event={"ID":"34ee2aba-8005-48b9-9cd8-459efcc498c5","Type":"ContainerStarted","Data":"f7666e00c59de3eb7ea878effbe89560307f9169f101e43f7c9b9b4bbaad3baf"}
Oct 31 23:17:01.261520 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:01.261527    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10903-01497c" event={"ID":"34ee2aba-8005-48b9-9cd8-459efcc498c5","Type":"ContainerStarted","Data":"02817dcd58f8a67eb5f5e7019b858b2925e0cb4dbf69a7c6a395578d4eacd1d9"}
Oct 31 23:17:01.278280 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:01.278207    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-10903-01497c" podStartSLOduration=2.278188718 podStartE2EDuration="2.278188718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:16:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:17:01.277624383 +0000 UTC m=+1640.024813887" watchObservedRunningTime="2026-10-31 23:17:01.278188718 +0000 UTC m=+1640.025378247"
Oct 31 23:17:02.435709 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:02.435677    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:17:02.522200 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:02.522165    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10903-01497c"]
Oct 31 23:17:02.728078 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:02.632507    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10903-01497c"]
Oct 31 23:17:02.825793 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:02.825762    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10903-2c89df"]
Oct 31 23:17:02.826961 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:02.826934    4301 cpu_manager.go:414] "containerMap: removing container" podUID="34ee2aba-8005-48b9-9cd8-459efcc498c5" containerName="test-curl-10903-01497c"
Oct 31 23:17:02.827130 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:02.827115    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34ee2aba-8005-48b9-9cd8-459efcc498c5" containerName="test-curl-10903-01497c"
Oct 31 23:17:02.827411 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:02.827393    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="34ee2aba-8005-48b9-9cd8-459efcc498c5" containerName="test-curl-10903-01497c"
Oct 31 23:17:02.837793 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:02.837766    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10903-2c89df"
Oct 31 23:17:02.843060 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:02.843034    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10903-2c89df"]
Oct 31 23:17:02.871709 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:17:02.871659    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:17:02.938865 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:02.938829    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcnwh\" (UniqueName: \"kubernetes.io/projected/62f6fc65-c368-4f46-b12a-6b4becd17fc7-kube-api-access-hcnwh\") pod \"test-curl-10903-2c89df\" (UID: \"62f6fc65-c368-4f46-b12a-6b4becd17fc7\") " pod="opendatahub/test-curl-10903-2c89df"
Oct 31 23:17:03.040223 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:03.040140    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcnwh\" (UniqueName: \"kubernetes.io/projected/62f6fc65-c368-4f46-b12a-6b4becd17fc7-kube-api-access-hcnwh\") pod \"test-curl-10903-2c89df\" (UID: \"62f6fc65-c368-4f46-b12a-6b4becd17fc7\") " pod="opendatahub/test-curl-10903-2c89df"
Oct 31 23:17:03.055223 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:03.055188    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcnwh\" (UniqueName: \"kubernetes.io/projected/62f6fc65-c368-4f46-b12a-6b4becd17fc7-kube-api-access-hcnwh\") pod \"test-curl-10903-2c89df\" (UID: \"62f6fc65-c368-4f46-b12a-6b4becd17fc7\") " pod="opendatahub/test-curl-10903-2c89df"
Oct 31 23:17:03.171334 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:03.171297    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10903-2c89df"
Oct 31 23:17:03.300119 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:03.300034    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-10903-01497c" podUID="34ee2aba-8005-48b9-9cd8-459efcc498c5" containerName="test-curl-10903-01497c" containerID="cri-o://f7666e00c59de3eb7ea878effbe89560307f9169f101e43f7c9b9b4bbaad3baf" gracePeriod=2
Oct 31 23:17:03.301879 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:17:03.301851    4301 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10903-01497c\" is forbidden: User \"system:node:ip-10-0-135-163.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-163.us-east-2.compute.internal' and this object" podUID="34ee2aba-8005-48b9-9cd8-459efcc498c5" pod="opendatahub/test-curl-10903-01497c"
Oct 31 23:17:03.590434 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:03.588020    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10903-2c89df"]
Oct 31 23:17:03.707137 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:17:03.707080    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62f6fc65_c368_4f46_b12a_6b4becd17fc7.slice/crio-9b1026d110714822354600effb9f2ebaa729234dff33100fa2510169b33055bd WatchSource:0}: Error finding container 9b1026d110714822354600effb9f2ebaa729234dff33100fa2510169b33055bd: Status 404 returned error can't find the container with id 9b1026d110714822354600effb9f2ebaa729234dff33100fa2510169b33055bd
Oct 31 23:17:04.361293 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:04.355971    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10903-2c89df" event={"ID":"62f6fc65-c368-4f46-b12a-6b4becd17fc7","Type":"ContainerStarted","Data":"14d5f0bca143a34a8574dc5db229c8aaea53c17f9425bca877b90d5b7bbe0576"}
Oct 31 23:17:04.361293 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:04.356046    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10903-2c89df" event={"ID":"62f6fc65-c368-4f46-b12a-6b4becd17fc7","Type":"ContainerStarted","Data":"9b1026d110714822354600effb9f2ebaa729234dff33100fa2510169b33055bd"}
Oct 31 23:17:04.389378 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:04.384029    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-10903-2c89df" podStartSLOduration=2.384008598 podStartE2EDuration="2.384008598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:17:04.378115067 +0000 UTC m=+1643.125304574" watchObservedRunningTime="2026-10-31 23:17:04.384008598 +0000 UTC m=+1643.131198104"
Oct 31 23:17:06.152873 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:06.152687    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10903-01497c"
Oct 31 23:17:06.213402 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:06.213320    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34ee2aba-8005-48b9-9cd8-459efcc498c5-kube-api-access-gbd4f\" (UniqueName: \"kubernetes.io/projected/34ee2aba-8005-48b9-9cd8-459efcc498c5-kube-api-access-gbd4f\") pod \"34ee2aba-8005-48b9-9cd8-459efcc498c5\" (UID: \"34ee2aba-8005-48b9-9cd8-459efcc498c5\") "
Oct 31 23:17:06.218514 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:06.218464    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ee2aba-8005-48b9-9cd8-459efcc498c5-kube-api-access-gbd4f" pod "34ee2aba-8005-48b9-9cd8-459efcc498c5" (UID: "34ee2aba-8005-48b9-9cd8-459efcc498c5"). InnerVolumeSpecName "kube-api-access-gbd4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:17:06.320900 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:06.320805    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbd4f\" (UniqueName: \"kubernetes.io/projected/34ee2aba-8005-48b9-9cd8-459efcc498c5-kube-api-access-gbd4f\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:17:06.442380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:06.441644    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10903-2c89df"]
Oct 31 23:17:06.462382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:06.458509    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10903-2c89df"]
Oct 31 23:17:06.515383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:06.512818    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10903-e66968"]
Oct 31 23:17:06.515383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:06.513880    4301 cpu_manager.go:414] "containerMap: removing container" podUID="62f6fc65-c368-4f46-b12a-6b4becd17fc7" containerName="test-curl-10903-2c89df"
Oct 31 23:17:06.515383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:06.513903    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62f6fc65-c368-4f46-b12a-6b4becd17fc7" containerName="test-curl-10903-2c89df"
Oct 31 23:17:06.515383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:06.514051    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="62f6fc65-c368-4f46-b12a-6b4becd17fc7" containerName="test-curl-10903-2c89df"
Oct 31 23:17:06.588455 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:06.551879    4301 generic.go:356] "Generic (PLEG): container finished" podID="34ee2aba-8005-48b9-9cd8-459efcc498c5" containerID="f7666e00c59de3eb7ea878effbe89560307f9169f101e43f7c9b9b4bbaad3baf" exitCode=137
Oct 31 23:17:06.588455 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:06.552078    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-10903-2c89df" podUID="62f6fc65-c368-4f46-b12a-6b4becd17fc7" containerName="test-curl-10903-2c89df" containerID="cri-o://14d5f0bca143a34a8574dc5db229c8aaea53c17f9425bca877b90d5b7bbe0576" gracePeriod=2
Oct 31 23:17:06.588455 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:06.552530    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10903-01497c"
Oct 31 23:17:06.588455 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:17:06.558639    4301 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10903-2c89df\" is forbidden: User \"system:node:ip-10-0-135-163.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-163.us-east-2.compute.internal' and this object" podUID="62f6fc65-c368-4f46-b12a-6b4becd17fc7" pod="opendatahub/test-curl-10903-2c89df"
Oct 31 23:17:06.588455 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:17:06.561825    4301 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10903-2c89df\" is forbidden: User \"system:node:ip-10-0-135-163.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-163.us-east-2.compute.internal' and this object" podUID="62f6fc65-c368-4f46-b12a-6b4becd17fc7" pod="opendatahub/test-curl-10903-2c89df"
Oct 31 23:17:06.588455 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:06.587486    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10903-e66968"
Oct 31 23:17:06.589300 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:06.588557    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10903-e66968"]
Oct 31 23:17:06.589300 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:06.588605    4301 scope.go:122] "RemoveContainer" containerID="f7666e00c59de3eb7ea878effbe89560307f9169f101e43f7c9b9b4bbaad3baf"
Oct 31 23:17:06.626805 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:17:06.612005    4301 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10903-2c89df\" is forbidden: User \"system:node:ip-10-0-135-163.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-163.us-east-2.compute.internal' and this object" podUID="62f6fc65-c368-4f46-b12a-6b4becd17fc7" pod="opendatahub/test-curl-10903-2c89df"
Oct 31 23:17:06.706250 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:06.660681    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blg82\" (UniqueName: \"kubernetes.io/projected/1bbadda6-8512-4118-9eab-cd2e7c808b8b-kube-api-access-blg82\") pod \"test-curl-10903-e66968\" (UID: \"1bbadda6-8512-4118-9eab-cd2e7c808b8b\") " pod="opendatahub/test-curl-10903-e66968"
Oct 31 23:17:06.765803 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:06.765769    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blg82\" (UniqueName: \"kubernetes.io/projected/1bbadda6-8512-4118-9eab-cd2e7c808b8b-kube-api-access-blg82\") pod \"test-curl-10903-e66968\" (UID: \"1bbadda6-8512-4118-9eab-cd2e7c808b8b\") " pod="opendatahub/test-curl-10903-e66968"
Oct 31 23:17:06.766056 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:17:06.765944    4301 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10903-01497c\" is forbidden: User \"system:node:ip-10-0-135-163.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-163.us-east-2.compute.internal' and this object" podUID="34ee2aba-8005-48b9-9cd8-459efcc498c5" pod="opendatahub/test-curl-10903-01497c"
Oct 31 23:17:06.772620 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:17:06.772581    4301 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10903-2c89df\" is forbidden: User \"system:node:ip-10-0-135-163.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-163.us-east-2.compute.internal' and this object" podUID="62f6fc65-c368-4f46-b12a-6b4becd17fc7" pod="opendatahub/test-curl-10903-2c89df"
Oct 31 23:17:06.799415 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:06.796142    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blg82\" (UniqueName: \"kubernetes.io/projected/1bbadda6-8512-4118-9eab-cd2e7c808b8b-kube-api-access-blg82\") pod \"test-curl-10903-e66968\" (UID: \"1bbadda6-8512-4118-9eab-cd2e7c808b8b\") " pod="opendatahub/test-curl-10903-e66968"
Oct 31 23:17:06.800274 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:06.800251    4301 scope.go:122] "RemoveContainer" containerID="f7666e00c59de3eb7ea878effbe89560307f9169f101e43f7c9b9b4bbaad3baf"
Oct 31 23:17:06.802588 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:17:06.802218    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7666e00c59de3eb7ea878effbe89560307f9169f101e43f7c9b9b4bbaad3baf\": container with ID starting with f7666e00c59de3eb7ea878effbe89560307f9169f101e43f7c9b9b4bbaad3baf not found: ID does not exist" containerID="f7666e00c59de3eb7ea878effbe89560307f9169f101e43f7c9b9b4bbaad3baf"
Oct 31 23:17:06.802588 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:06.802277    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7666e00c59de3eb7ea878effbe89560307f9169f101e43f7c9b9b4bbaad3baf"} err="failed to get container status \"f7666e00c59de3eb7ea878effbe89560307f9169f101e43f7c9b9b4bbaad3baf\": rpc error: code = NotFound desc = could not find container \"f7666e00c59de3eb7ea878effbe89560307f9169f101e43f7c9b9b4bbaad3baf\": container with ID starting with f7666e00c59de3eb7ea878effbe89560307f9169f101e43f7c9b9b4bbaad3baf not found: ID does not exist"
Oct 31 23:17:07.082420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:07.082065    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10903-e66968"
Oct 31 23:17:07.459494 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:07.459457    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34ee2aba-8005-48b9-9cd8-459efcc498c5" path="/var/lib/kubelet/pods/34ee2aba-8005-48b9-9cd8-459efcc498c5/volumes"
Oct 31 23:17:07.626322 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:07.626284    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"]
Oct 31 23:17:07.668899 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:07.668863    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:17:07.712579 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:07.711475    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5zt2\" (UniqueName: \"kubernetes.io/projected/71069228-75f3-453b-92b7-2648a3921d39-kube-api-access-b5zt2\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:17:07.712579 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:07.711618    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71069228-75f3-453b-92b7-2648a3921d39-istio-podinfo\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:17:07.712579 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:07.711684    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-workload-certs\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:17:07.712579 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:07.711721    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-credential-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:17:07.712579 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:07.711862    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71069228-75f3-453b-92b7-2648a3921d39-istio-token\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:17:07.712579 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:07.711907    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-istio-envoy\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:17:07.712579 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:07.711949    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-istio-data\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:17:07.712579 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:07.711984    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71069228-75f3-453b-92b7-2648a3921d39-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:17:07.712579 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:07.712042    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-workload-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:17:07.862110 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:07.825259    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71069228-75f3-453b-92b7-2648a3921d39-istio-token\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:17:07.862110 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:07.825329    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-istio-envoy\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:17:07.862110 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:07.825391    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-istio-data\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:17:07.862110 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:07.825424    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71069228-75f3-453b-92b7-2648a3921d39-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:17:07.862110 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:07.825485    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-workload-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:17:07.862110 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:07.825586    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5zt2\" (UniqueName: \"kubernetes.io/projected/71069228-75f3-453b-92b7-2648a3921d39-kube-api-access-b5zt2\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:17:07.862110 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:07.825715    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71069228-75f3-453b-92b7-2648a3921d39-istio-podinfo\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:17:07.862110 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:07.825781    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-workload-certs\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:17:07.862110 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:07.825831    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-credential-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:17:07.862110 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:07.826519    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-credential-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:17:07.862110 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:07.826819    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-workload-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:17:07.862110 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:07.826968    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71069228-75f3-453b-92b7-2648a3921d39-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:17:07.862110 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:07.829884    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-istio-envoy\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:17:07.862110 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:07.830132    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-istio-data\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:17:07.862110 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:07.830373    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-workload-certs\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:17:07.862110 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:07.835395    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"]
Oct 31 23:17:07.862110 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:07.842903    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:17:08.071416 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.067606    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5zt2\" (UniqueName: \"kubernetes.io/projected/71069228-75f3-453b-92b7-2648a3921d39-kube-api-access-b5zt2\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:17:08.109915 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.109617    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-credential-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:17:08.109915 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.109654    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71069228-75f3-453b-92b7-2648a3921d39-istio-podinfo\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:17:08.109915 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.109676    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-workload-certs\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:17:08.109915 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.109887    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:17:08.110250 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.109990    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9zm8\" (UniqueName: \"kubernetes.io/projected/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-kube-api-access-p9zm8\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:17:08.110250 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.110052    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-workload-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:17:08.110250 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.110081    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istio-token\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:17:08.110250 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.110121    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istio-data\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:17:08.110250 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.110172    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istio-envoy\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:17:08.110520 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.110272    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istio-podinfo\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:17:08.245990 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.243021    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71069228-75f3-453b-92b7-2648a3921d39-istio-token\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:17:08.330778 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.320006    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"]
Oct 31 23:17:08.333548 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:17:08.332022    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bbadda6_8512_4118_9eab_cd2e7c808b8b.slice/crio-e581d6f393ae65d411197d308633c27dd12325d2e665de1c2d165d15915ccd97 WatchSource:0}: Error finding container e581d6f393ae65d411197d308633c27dd12325d2e665de1c2d165d15915ccd97: Status 404 returned error can't find the container with id e581d6f393ae65d411197d308633c27dd12325d2e665de1c2d165d15915ccd97
Oct 31 23:17:08.347508 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.347473    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"]
Oct 31 23:17:08.357514 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.357476    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-workload-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:17:08.357703 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.357532    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istio-token\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:17:08.357703 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.357570    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istio-data\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:17:08.357703 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.357625    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istio-envoy\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:17:08.357854 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.357705    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istio-podinfo\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:17:08.357854 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.357839    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-credential-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:17:08.357949 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.357868    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-workload-certs\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:17:08.358058 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.358038    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:17:08.358119 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.358106    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9zm8\" (UniqueName: \"kubernetes.io/projected/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-kube-api-access-p9zm8\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:17:08.358819 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.358792    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-workload-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:17:08.360844 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.360823    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10903-e66968"]
Oct 31 23:17:08.362030 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.362000    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:17:08.365185 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.365161    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istio-data\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:17:08.398121 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.398094    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-credential-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:17:08.411477 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.409149    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:17:08.411477 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.409434    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-workload-certs\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:17:08.453394 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.446043    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:17:08.503435 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.479868    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9zm8\" (UniqueName: \"kubernetes.io/projected/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-kube-api-access-p9zm8\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:17:08.559846 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.559500    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istio-podinfo\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:17:08.572867 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.572836    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istio-envoy\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:17:08.574468 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.574428    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istio-token\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:17:08.891443 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:08.866538    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:17:09.058542 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:09.058493    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:17:09.073503 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:09.072138    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:17:09.309399 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:09.298119    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10903-e66968" event={"ID":"1bbadda6-8512-4118-9eab-cd2e7c808b8b","Type":"ContainerStarted","Data":"e581d6f393ae65d411197d308633c27dd12325d2e665de1c2d165d15915ccd97"}
Oct 31 23:17:09.912763 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:09.862694    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"]
Oct 31 23:17:10.165452 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:10.161505    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:17:10.165452 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:10.161691    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:17:10.165452 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:10.161821    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:17:10.590987 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:10.570691    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10903-e66968" event={"ID":"1bbadda6-8512-4118-9eab-cd2e7c808b8b","Type":"ContainerStarted","Data":"c81cd561918b68fc9d781ac3fb32ad155d82bff5bd371f421fa270f4c6395f84"}
Oct 31 23:17:10.590987 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:10.584201    4301 generic.go:356] "Generic (PLEG): container finished" podID="62f6fc65-c368-4f46-b12a-6b4becd17fc7" containerID="14d5f0bca143a34a8574dc5db229c8aaea53c17f9425bca877b90d5b7bbe0576" exitCode=137
Oct 31 23:17:10.590987 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:10.588578    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk" event={"ID":"71069228-75f3-453b-92b7-2648a3921d39","Type":"ContainerStarted","Data":"0335f0b13d441e29ffad3a6ab074f9c63f3db2270deb955536b31fac461f9e34"}
Oct 31 23:17:10.601990 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:10.593919    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10903-2c89df"
Oct 31 23:17:10.691477 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:10.669008    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62f6fc65-c368-4f46-b12a-6b4becd17fc7-kube-api-access-hcnwh\" (UniqueName: \"kubernetes.io/projected/62f6fc65-c368-4f46-b12a-6b4becd17fc7-kube-api-access-hcnwh\") pod \"62f6fc65-c368-4f46-b12a-6b4becd17fc7\" (UID: \"62f6fc65-c368-4f46-b12a-6b4becd17fc7\") "
Oct 31 23:17:10.893414 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:10.892026    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62f6fc65-c368-4f46-b12a-6b4becd17fc7-kube-api-access-hcnwh" pod "62f6fc65-c368-4f46-b12a-6b4becd17fc7" (UID: "62f6fc65-c368-4f46-b12a-6b4becd17fc7"). InnerVolumeSpecName "kube-api-access-hcnwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:17:10.893414 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:10.893206    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-10903-e66968" podStartSLOduration=4.893186805 podStartE2EDuration="4.893186805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:17:10.613500768 +0000 UTC m=+1649.360690278" watchObservedRunningTime="2026-10-31 23:17:10.893186805 +0000 UTC m=+1649.640376314"
Oct 31 23:17:10.901466 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:10.901181    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcnwh\" (UniqueName: \"kubernetes.io/projected/62f6fc65-c368-4f46-b12a-6b4becd17fc7-kube-api-access-hcnwh\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:17:10.921726 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:10.921696    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"]
Oct 31 23:17:10.943580 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:10.942696    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:17:10.943580 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:10.942822    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:17:10.943580 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:10.942863    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:17:11.803376 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:11.780525    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62f6fc65-c368-4f46-b12a-6b4becd17fc7" path="/var/lib/kubelet/pods/62f6fc65-c368-4f46-b12a-6b4becd17fc7/volumes"
Oct 31 23:17:11.823403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:11.814854    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk" podStartSLOduration=4.814831928 podStartE2EDuration="4.814831928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:17:11.813859131 +0000 UTC m=+1650.561048660" watchObservedRunningTime="2026-10-31 23:17:11.814831928 +0000 UTC m=+1650.562021435"
Oct 31 23:17:11.823403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:11.821321    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10903-2c89df"
Oct 31 23:17:11.870449 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:11.804485    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk" event={"ID":"71069228-75f3-453b-92b7-2648a3921d39","Type":"ContainerStarted","Data":"ae5c3f76dc49255504c1d601a5010fcfb01fa885c32cbe2f78f3eb1192d38c27"}
Oct 31 23:17:11.870694 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:11.870669    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f" event={"ID":"f8595a54-27b2-43a5-9e7d-f7d0fd41c523","Type":"ContainerStarted","Data":"cc3499284872061224ffd6f190018441fa71576d01dc5c1c344cc6425e0ca619"}
Oct 31 23:17:11.870819 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:11.870805    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f" event={"ID":"f8595a54-27b2-43a5-9e7d-f7d0fd41c523","Type":"ContainerStarted","Data":"38365cc7d62a343d5856a2c22523885615a5885bff469f2804757bfb2d96ffb7"}
Oct 31 23:17:11.870962 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:11.870949    4301 scope.go:122] "RemoveContainer" containerID="14d5f0bca143a34a8574dc5db229c8aaea53c17f9425bca877b90d5b7bbe0576"
Oct 31 23:17:11.871458 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:11.871404    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f" podStartSLOduration=4.871386962 podStartE2EDuration="4.871386962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:17:11.870695345 +0000 UTC m=+1650.617884850" watchObservedRunningTime="2026-10-31 23:17:11.871386962 +0000 UTC m=+1650.618576469"
Oct 31 23:17:11.893457 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:11.893429    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:17:12.064542 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:12.064289    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10903-e66968"]
Oct 31 23:17:12.070856 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:12.070823    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10903-e66968"]
Oct 31 23:17:12.316558 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:12.313273    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10903-677e04"]
Oct 31 23:17:12.316558 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:12.314261    4301 cpu_manager.go:414] "containerMap: removing container" podUID="1bbadda6-8512-4118-9eab-cd2e7c808b8b" containerName="test-curl-10903-e66968"
Oct 31 23:17:12.316558 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:12.314280    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bbadda6-8512-4118-9eab-cd2e7c808b8b" containerName="test-curl-10903-e66968"
Oct 31 23:17:12.316558 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:12.314450    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bbadda6-8512-4118-9eab-cd2e7c808b8b" containerName="test-curl-10903-e66968"
Oct 31 23:17:12.322261 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:12.319407    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10903-677e04"]
Oct 31 23:17:12.322261 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:12.319490    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10903-677e04"
Oct 31 23:17:12.411846 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:12.411815    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hq4s\" (UniqueName: \"kubernetes.io/projected/73cd1a44-54ba-4fc6-bb4a-357b61569ecb-kube-api-access-9hq4s\") pod \"test-curl-10903-677e04\" (UID: \"73cd1a44-54ba-4fc6-bb4a-357b61569ecb\") " pod="opendatahub/test-curl-10903-677e04"
Oct 31 23:17:12.459987 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:12.459957    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:17:12.460266 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:12.460252    4301 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 31 23:17:12.460426 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:12.460391    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk" podUID="71069228-75f3-453b-92b7-2648a3921d39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 31 23:17:12.513299 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:12.513266    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hq4s\" (UniqueName: \"kubernetes.io/projected/73cd1a44-54ba-4fc6-bb4a-357b61569ecb-kube-api-access-9hq4s\") pod \"test-curl-10903-677e04\" (UID: \"73cd1a44-54ba-4fc6-bb4a-357b61569ecb\") " pod="opendatahub/test-curl-10903-677e04"
Oct 31 23:17:12.525166 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:12.525135    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hq4s\" (UniqueName: \"kubernetes.io/projected/73cd1a44-54ba-4fc6-bb4a-357b61569ecb-kube-api-access-9hq4s\") pod \"test-curl-10903-677e04\" (UID: \"73cd1a44-54ba-4fc6-bb4a-357b61569ecb\") " pod="opendatahub/test-curl-10903-677e04"
Oct 31 23:17:12.673223 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:12.673186    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10903-677e04"
Oct 31 23:17:12.857382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:12.856993    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-10903-e66968" podUID="1bbadda6-8512-4118-9eab-cd2e7c808b8b" containerName="test-curl-10903-e66968" containerID="cri-o://c81cd561918b68fc9d781ac3fb32ad155d82bff5bd371f421fa270f4c6395f84" gracePeriod=2
Oct 31 23:17:12.858882 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:17:12.858860    4301 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10903-e66968\" is forbidden: User \"system:node:ip-10-0-135-163.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-163.us-east-2.compute.internal' and this object" podUID="1bbadda6-8512-4118-9eab-cd2e7c808b8b" pod="opendatahub/test-curl-10903-e66968"
Oct 31 23:17:12.919429 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:12.918258    4301 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 23:17:12.919429 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:12.918321    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f" podUID="f8595a54-27b2-43a5-9e7d-f7d0fd41c523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 23:17:13.035385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:13.032178    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10903-677e04"]
Oct 31 23:17:13.466531 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:13.466048    4301 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 31 23:17:13.466531 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:13.466117    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk" podUID="71069228-75f3-453b-92b7-2648a3921d39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 31 23:17:13.900396 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:13.899040    4301 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 23:17:13.900396 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:13.899099    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f" podUID="f8595a54-27b2-43a5-9e7d-f7d0fd41c523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 23:17:13.911544 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:13.904117    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10903-677e04" event={"ID":"73cd1a44-54ba-4fc6-bb4a-357b61569ecb","Type":"ContainerStarted","Data":"d66c89413b3dc74469f95a1bd4f12754ac140c5c996d98d1064c5a94da098f96"}
Oct 31 23:17:13.911544 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:13.904165    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10903-677e04" event={"ID":"73cd1a44-54ba-4fc6-bb4a-357b61569ecb","Type":"ContainerStarted","Data":"3f6ddbcb3b1fbdcdd76ed634a36056f6bc7e2880a377ee467b2f7b1b0dd467c2"}
Oct 31 23:17:14.465382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:14.464711    4301 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 31 23:17:14.465382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:14.464768    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk" podUID="71069228-75f3-453b-92b7-2648a3921d39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 31 23:17:14.748524 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:14.729215    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-10903-677e04" podStartSLOduration=2.7291923909999998 podStartE2EDuration="2.729192391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:17:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:17:13.938565581 +0000 UTC m=+1652.685755087" watchObservedRunningTime="2026-10-31 23:17:14.729192391 +0000 UTC m=+1653.476381899"
Oct 31 23:17:14.756387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:14.756040    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10903-677e04"]
Oct 31 23:17:14.827387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:14.825442    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10903-677e04"]
Oct 31 23:17:14.923845 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:14.922714    4301 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 23:17:14.923845 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:14.922776    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f" podUID="f8595a54-27b2-43a5-9e7d-f7d0fd41c523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 23:17:15.668464 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:15.605596    4301 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 31 23:17:15.668464 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:15.605669    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk" podUID="71069228-75f3-453b-92b7-2648a3921d39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 31 23:17:15.903256 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:15.902453    4301 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 23:17:15.903256 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:15.902504    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f" podUID="f8595a54-27b2-43a5-9e7d-f7d0fd41c523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 23:17:15.913122 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:15.913072    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10903-e66968"
Oct 31 23:17:15.954449 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:15.953582    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bbadda6-8512-4118-9eab-cd2e7c808b8b-kube-api-access-blg82\" (UniqueName: \"kubernetes.io/projected/1bbadda6-8512-4118-9eab-cd2e7c808b8b-kube-api-access-blg82\") pod \"1bbadda6-8512-4118-9eab-cd2e7c808b8b\" (UID: \"1bbadda6-8512-4118-9eab-cd2e7c808b8b\") "
Oct 31 23:17:15.972225 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:15.972166    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bbadda6-8512-4118-9eab-cd2e7c808b8b-kube-api-access-blg82" pod "1bbadda6-8512-4118-9eab-cd2e7c808b8b" (UID: "1bbadda6-8512-4118-9eab-cd2e7c808b8b"). InnerVolumeSpecName "kube-api-access-blg82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:17:15.999925 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:15.999754    4301 generic.go:356] "Generic (PLEG): container finished" podID="1bbadda6-8512-4118-9eab-cd2e7c808b8b" containerID="c81cd561918b68fc9d781ac3fb32ad155d82bff5bd371f421fa270f4c6395f84" exitCode=137
Oct 31 23:17:16.001387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:16.000178    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10903-e66968"
Oct 31 23:17:16.001712 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:16.001537    4301 scope.go:122] "RemoveContainer" containerID="c81cd561918b68fc9d781ac3fb32ad155d82bff5bd371f421fa270f4c6395f84"
Oct 31 23:17:16.001712 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:16.001695    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-10903-677e04" podUID="73cd1a44-54ba-4fc6-bb4a-357b61569ecb" containerName="test-curl-10903-677e04" containerID="cri-o://d66c89413b3dc74469f95a1bd4f12754ac140c5c996d98d1064c5a94da098f96" gracePeriod=2
Oct 31 23:17:16.019984 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:16.019327    4301 scope.go:122] "RemoveContainer" containerID="c81cd561918b68fc9d781ac3fb32ad155d82bff5bd371f421fa270f4c6395f84"
Oct 31 23:17:16.020101 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:17:16.020065    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c81cd561918b68fc9d781ac3fb32ad155d82bff5bd371f421fa270f4c6395f84\": container with ID starting with c81cd561918b68fc9d781ac3fb32ad155d82bff5bd371f421fa270f4c6395f84 not found: ID does not exist" containerID="c81cd561918b68fc9d781ac3fb32ad155d82bff5bd371f421fa270f4c6395f84"
Oct 31 23:17:16.020156 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:16.020093    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c81cd561918b68fc9d781ac3fb32ad155d82bff5bd371f421fa270f4c6395f84"} err="failed to get container status \"c81cd561918b68fc9d781ac3fb32ad155d82bff5bd371f421fa270f4c6395f84\": rpc error: code = NotFound desc = could not find container \"c81cd561918b68fc9d781ac3fb32ad155d82bff5bd371f421fa270f4c6395f84\": container with ID starting with c81cd561918b68fc9d781ac3fb32ad155d82bff5bd371f421fa270f4c6395f84 not found: ID does not exist"
Oct 31 23:17:16.021531 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:17:16.021499    4301 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10903-e66968\" is forbidden: User \"system:node:ip-10-0-135-163.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-163.us-east-2.compute.internal' and this object" podUID="1bbadda6-8512-4118-9eab-cd2e7c808b8b" pod="opendatahub/test-curl-10903-e66968"
Oct 31 23:17:16.076168 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:16.069913    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blg82\" (UniqueName: \"kubernetes.io/projected/1bbadda6-8512-4118-9eab-cd2e7c808b8b-kube-api-access-blg82\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:17:16.464388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:16.460505    4301 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 31 23:17:16.464388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:16.460592    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk" podUID="71069228-75f3-453b-92b7-2648a3921d39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 31 23:17:16.896213 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:16.896177    4301 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 23:17:16.896533 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:16.896504    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f" podUID="f8595a54-27b2-43a5-9e7d-f7d0fd41c523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 23:17:17.477794 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:17.477758    4301 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 31 23:17:17.478254 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:17.477816    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk" podUID="71069228-75f3-453b-92b7-2648a3921d39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 31 23:17:17.478616 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:17.478569    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bbadda6-8512-4118-9eab-cd2e7c808b8b" path="/var/lib/kubelet/pods/1bbadda6-8512-4118-9eab-cd2e7c808b8b/volumes"
Oct 31 23:17:17.894219 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:17.893755    4301 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 23:17:17.894219 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:17.893822    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f" podUID="f8595a54-27b2-43a5-9e7d-f7d0fd41c523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 23:17:18.427091 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:18.427046    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10903-677e04"
Oct 31 23:17:18.473383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:18.460791    4301 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 31 23:17:18.473383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:18.460853    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk" podUID="71069228-75f3-453b-92b7-2648a3921d39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 31 23:17:18.610653 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:18.610573    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73cd1a44-54ba-4fc6-bb4a-357b61569ecb-kube-api-access-9hq4s\" (UniqueName: \"kubernetes.io/projected/73cd1a44-54ba-4fc6-bb4a-357b61569ecb-kube-api-access-9hq4s\") pod \"73cd1a44-54ba-4fc6-bb4a-357b61569ecb\" (UID: \"73cd1a44-54ba-4fc6-bb4a-357b61569ecb\") "
Oct 31 23:17:18.636531 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:18.618576    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73cd1a44-54ba-4fc6-bb4a-357b61569ecb-kube-api-access-9hq4s" pod "73cd1a44-54ba-4fc6-bb4a-357b61569ecb" (UID: "73cd1a44-54ba-4fc6-bb4a-357b61569ecb"). InnerVolumeSpecName "kube-api-access-9hq4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:17:18.737421 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:18.728514    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hq4s\" (UniqueName: \"kubernetes.io/projected/73cd1a44-54ba-4fc6-bb4a-357b61569ecb-kube-api-access-9hq4s\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:17:18.904097 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:18.904066    4301 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 23:17:18.904453 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:18.904397    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f" podUID="f8595a54-27b2-43a5-9e7d-f7d0fd41c523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 23:17:19.106147 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:19.106110    4301 generic.go:356] "Generic (PLEG): container finished" podID="73cd1a44-54ba-4fc6-bb4a-357b61569ecb" containerID="d66c89413b3dc74469f95a1bd4f12754ac140c5c996d98d1064c5a94da098f96" exitCode=137
Oct 31 23:17:19.106147 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:19.106154    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10903-677e04"
Oct 31 23:17:19.106447 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:19.106218    4301 scope.go:122] "RemoveContainer" containerID="d66c89413b3dc74469f95a1bd4f12754ac140c5c996d98d1064c5a94da098f96"
Oct 31 23:17:19.118128 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:19.117854    4301 scope.go:122] "RemoveContainer" containerID="d66c89413b3dc74469f95a1bd4f12754ac140c5c996d98d1064c5a94da098f96"
Oct 31 23:17:19.118417 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:17:19.118323    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d66c89413b3dc74469f95a1bd4f12754ac140c5c996d98d1064c5a94da098f96\": container with ID starting with d66c89413b3dc74469f95a1bd4f12754ac140c5c996d98d1064c5a94da098f96 not found: ID does not exist" containerID="d66c89413b3dc74469f95a1bd4f12754ac140c5c996d98d1064c5a94da098f96"
Oct 31 23:17:19.118417 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:19.118381    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d66c89413b3dc74469f95a1bd4f12754ac140c5c996d98d1064c5a94da098f96"} err="failed to get container status \"d66c89413b3dc74469f95a1bd4f12754ac140c5c996d98d1064c5a94da098f96\": rpc error: code = NotFound desc = could not find container \"d66c89413b3dc74469f95a1bd4f12754ac140c5c996d98d1064c5a94da098f96\": container with ID starting with d66c89413b3dc74469f95a1bd4f12754ac140c5c996d98d1064c5a94da098f96 not found: ID does not exist"
Oct 31 23:17:19.120210 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:17:19.120184    4301 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-10903-677e04\" is forbidden: User \"system:node:ip-10-0-135-163.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-163.us-east-2.compute.internal' and this object" podUID="73cd1a44-54ba-4fc6-bb4a-357b61569ecb" pod="opendatahub/test-curl-10903-677e04"
Oct 31 23:17:19.423154 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:19.422493    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73cd1a44-54ba-4fc6-bb4a-357b61569ecb" path="/var/lib/kubelet/pods/73cd1a44-54ba-4fc6-bb4a-357b61569ecb/volumes"
Oct 31 23:17:19.473059 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:19.468215    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:17:19.473059 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:19.469185    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:17:19.473059 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:19.471399    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:17:20.214251 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:20.209238    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:17:20.214251 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:20.210906    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:17:20.214853 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:20.214476    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:17:20.626378 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:20.624427    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"]
Oct 31 23:17:20.626378 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:20.625314    4301 cpu_manager.go:414] "containerMap: removing container" podUID="73cd1a44-54ba-4fc6-bb4a-357b61569ecb" containerName="test-curl-10903-677e04"
Oct 31 23:17:20.626378 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:20.625334    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73cd1a44-54ba-4fc6-bb4a-357b61569ecb" containerName="test-curl-10903-677e04"
Oct 31 23:17:20.626378 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:20.625567    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="73cd1a44-54ba-4fc6-bb4a-357b61569ecb" containerName="test-curl-10903-677e04"
Oct 31 23:17:20.632607 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:20.632586    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"
Oct 31 23:17:20.644732 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:20.644697    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"]
Oct 31 23:17:20.645030 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:20.645004    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-36a15fc5\"/\"openshift-service-ca.crt\""
Oct 31 23:17:20.645150 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:20.645133    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-36a15fc5\"/\"default-dockercfg-q9qfn\""
Oct 31 23:17:20.645272 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:20.645253    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-36a15fc5\"/\"e2e-lifecycle-model-36a15fc5-kserve-self-signed-certs\""
Oct 31 23:17:20.645399 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:20.645371    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-36a15fc5\"/\"kube-root-ca.crt\""
Oct 31 23:17:20.716119 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:20.716087    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-dshm\") pod \"e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97\" (UID: \"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0\") " pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"
Oct 31 23:17:20.716119 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:20.716122    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-tmp-dir\") pod \"e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97\" (UID: \"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0\") " pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"
Oct 31 23:17:20.716431 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:20.716141    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-tls-certs\") pod \"e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97\" (UID: \"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0\") " pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"
Oct 31 23:17:20.716431 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:20.716161    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-home\") pod \"e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97\" (UID: \"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0\") " pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"
Oct 31 23:17:20.716431 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:20.716353    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlxx9\" (UniqueName: \"kubernetes.io/projected/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-kube-api-access-zlxx9\") pod \"e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97\" (UID: \"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0\") " pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"
Oct 31 23:17:20.716610 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:20.716471    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-model-cache\") pod \"e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97\" (UID: \"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0\") " pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"
Oct 31 23:17:20.818475 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:20.818432    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlxx9\" (UniqueName: \"kubernetes.io/projected/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-kube-api-access-zlxx9\") pod \"e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97\" (UID: \"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0\") " pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"
Oct 31 23:17:20.818671 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:20.818485    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-model-cache\") pod \"e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97\" (UID: \"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0\") " pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"
Oct 31 23:17:20.818671 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:20.818643    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-dshm\") pod \"e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97\" (UID: \"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0\") " pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"
Oct 31 23:17:20.818671 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:20.818667    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-tmp-dir\") pod \"e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97\" (UID: \"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0\") " pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"
Oct 31 23:17:20.818850 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:20.818683    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-tls-certs\") pod \"e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97\" (UID: \"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0\") " pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"
Oct 31 23:17:20.818850 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:20.818707    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-home\") pod \"e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97\" (UID: \"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0\") " pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"
Oct 31 23:17:20.819304 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:20.818978    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-model-cache\") pod \"e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97\" (UID: \"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0\") " pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"
Oct 31 23:17:20.819304 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:20.819054    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-tmp-dir\") pod \"e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97\" (UID: \"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0\") " pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"
Oct 31 23:17:20.819304 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:20.819129    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-home\") pod \"e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97\" (UID: \"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0\") " pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"
Oct 31 23:17:20.821502 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:20.821470    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-tls-certs\") pod \"e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97\" (UID: \"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0\") " pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"
Oct 31 23:17:20.821633 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:20.821599    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-dshm\") pod \"e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97\" (UID: \"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0\") " pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"
Oct 31 23:17:20.830044 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:20.830021    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlxx9\" (UniqueName: \"kubernetes.io/projected/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-kube-api-access-zlxx9\") pod \"e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97\" (UID: \"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0\") " pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"
Oct 31 23:17:20.988460 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:20.987769    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"
Oct 31 23:17:22.112504 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:22.112473    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"]
Oct 31 23:17:22.114534 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:17:22.114507    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2f530bc_ac7a_4ce3_96f1_fdddb435ace0.slice/crio-3a4c9624581e2ad717a2326c97e10bef08f734a2d742863c944d7ad3cfd1fc68 WatchSource:0}: Error finding container 3a4c9624581e2ad717a2326c97e10bef08f734a2d742863c944d7ad3cfd1fc68: Status 404 returned error can't find the container with id 3a4c9624581e2ad717a2326c97e10bef08f734a2d742863c944d7ad3cfd1fc68
Oct 31 23:17:22.216731 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:22.216687    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97" event={"ID":"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0","Type":"ContainerStarted","Data":"2fd72b7805b68199659be0e62969013b8900b097fe1fd251420d82d982d313ed"}
Oct 31 23:17:22.216914 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:22.216743    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97" event={"ID":"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0","Type":"ContainerStarted","Data":"3a4c9624581e2ad717a2326c97e10bef08f734a2d742863c944d7ad3cfd1fc68"}
Oct 31 23:17:22.238864 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:22.238799    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97" podStartSLOduration=2.2387852759999998 podStartE2EDuration="2.238785276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:17:22.234807864 +0000 UTC m=+1660.981997380" watchObservedRunningTime="2026-10-31 23:17:22.238785276 +0000 UTC m=+1660.985974781"
Oct 31 23:17:22.578393 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:22.572379    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 23:17:22.578393 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:22.572631    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx" podUID="9a58ab83-2394-48a8-aa4f-04c22ee67a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 23:17:23.002418 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:23.002387    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"
Oct 31 23:17:23.244401 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:23.243773    4301 generic.go:356] "Generic (PLEG): container finished" podID="9a58ab83-2394-48a8-aa4f-04c22ee67a24" containerID="d48357bb04b7e1e03a059677a5ad42bbccdf1017aba1ca95dad92fa740178e30" exitCode=0
Oct 31 23:17:23.248386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:23.244998    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx" event={"ID":"9a58ab83-2394-48a8-aa4f-04c22ee67a24","Type":"ContainerDied","Data":"d48357bb04b7e1e03a059677a5ad42bbccdf1017aba1ca95dad92fa740178e30"}
Oct 31 23:17:23.248386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:23.245623    4301 scope.go:122] "RemoveContainer" containerID="d48357bb04b7e1e03a059677a5ad42bbccdf1017aba1ca95dad92fa740178e30"
Oct 31 23:17:23.259003 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:23.252174    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:17:23.259003 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:23.252286    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:17:23.259003 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:23.252337    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:17:24.261252 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:24.261174    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx" event={"ID":"9a58ab83-2394-48a8-aa4f-04c22ee67a24","Type":"ContainerStarted","Data":"0b88e034f850cc37100ede8de632d0db0f238582eb0c3975b27dea1b6a615860"}
Oct 31 23:17:24.487383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:24.477952    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:17:24.487383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:24.478003    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 23:17:24.487383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:24.478065    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx" podUID="9a58ab83-2394-48a8-aa4f-04c22ee67a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 23:17:25.476062 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:25.476024    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 23:17:25.476513 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:25.476091    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx" podUID="9a58ab83-2394-48a8-aa4f-04c22ee67a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 23:17:26.476333 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:26.476298    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 23:17:26.476789 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:26.476390    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx" podUID="9a58ab83-2394-48a8-aa4f-04c22ee67a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 23:17:27.493995 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:27.493961    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 23:17:27.494599 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:27.494567    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx" podUID="9a58ab83-2394-48a8-aa4f-04c22ee67a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 23:17:28.475518 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:28.475487    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 23:17:28.475722 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:28.475544    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx" podUID="9a58ab83-2394-48a8-aa4f-04c22ee67a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 23:17:29.481778 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:29.481745    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:17:29.482512 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:29.482462    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:17:29.483687 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:29.483668    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:17:31.012385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:31.010683    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"
Oct 31 23:17:31.050424 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:31.050384    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"
Oct 31 23:17:31.406939 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:31.406912    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"
Oct 31 23:17:32.640112 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:17:32.640075    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:17:35.911671 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:35.910854    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"]
Oct 31 23:17:35.917704 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:35.917676    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"]
Oct 31 23:17:35.917846 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:35.917797    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:17:35.921748 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:35.921715    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-d9e90c-openshift-default-dockercfg-llv6m\""
Oct 31 23:17:36.040631 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:36.040582    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-workload-certs\") pod \"e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") " pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:17:36.047491 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:36.047448    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-istio-data\") pod \"e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") " pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:17:36.047781 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:36.047748    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/195909b1-e0bf-4679-84e3-50786cc7eaea-istio-token\") pod \"e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") " pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:17:36.047931 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:36.047915    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-istio-envoy\") pod \"e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") " pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:17:36.048180 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:36.048162    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-credential-socket\") pod \"e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") " pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:17:36.054634 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:36.054595    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-workload-socket\") pod \"e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") " pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:17:36.054901 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:36.054885    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/195909b1-e0bf-4679-84e3-50786cc7eaea-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") " pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:17:36.055006 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:36.054994    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/195909b1-e0bf-4679-84e3-50786cc7eaea-istio-podinfo\") pod \"e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") " pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:17:36.055094 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:36.055083    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v72j5\" (UniqueName: \"kubernetes.io/projected/195909b1-e0bf-4679-84e3-50786cc7eaea-kube-api-access-v72j5\") pod \"e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") " pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:17:36.156596 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:36.156556    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/195909b1-e0bf-4679-84e3-50786cc7eaea-istio-token\") pod \"e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") " pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:17:36.156922 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:36.156900    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-istio-envoy\") pod \"e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") " pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:17:36.157094 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:36.157079    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-credential-socket\") pod \"e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") " pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:17:36.157381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:36.157345    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-workload-socket\") pod \"e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") " pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:17:36.157636 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:36.157577    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/195909b1-e0bf-4679-84e3-50786cc7eaea-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") " pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:17:36.165932 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:36.157774    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/195909b1-e0bf-4679-84e3-50786cc7eaea-istio-podinfo\") pod \"e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") " pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:17:36.167777 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:36.166803    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-workload-socket\") pod \"e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") " pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:17:36.187934 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:36.167306    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-credential-socket\") pod \"e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") " pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:17:36.199239 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:36.167945    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v72j5\" (UniqueName: \"kubernetes.io/projected/195909b1-e0bf-4679-84e3-50786cc7eaea-kube-api-access-v72j5\") pod \"e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") " pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:17:36.206212 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:36.199680    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-workload-certs\") pod \"e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") " pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:17:36.206938 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:36.206891    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-istio-data\") pod \"e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") " pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:17:36.207616 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:36.167973    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/195909b1-e0bf-4679-84e3-50786cc7eaea-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") " pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:17:36.207759 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:36.206824    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-workload-certs\") pod \"e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") " pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:17:36.207867 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:36.189103    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-istio-envoy\") pod \"e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") " pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:17:36.207968 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:36.207417    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-istio-data\") pod \"e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") " pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:17:36.237213 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:36.237176    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/195909b1-e0bf-4679-84e3-50786cc7eaea-istio-podinfo\") pod \"e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") " pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:17:36.251459 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:36.240084    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v72j5\" (UniqueName: \"kubernetes.io/projected/195909b1-e0bf-4679-84e3-50786cc7eaea-kube-api-access-v72j5\") pod \"e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") " pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:17:36.260482 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:36.260451    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/195909b1-e0bf-4679-84e3-50786cc7eaea-istio-token\") pod \"e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") " pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:17:36.554198 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:36.554099    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:17:36.968109 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:36.967974    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"]
Oct 31 23:17:36.974716 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:36.974669    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:17:36.974828 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:36.974767    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:17:36.974897 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:36.974814    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:17:37.532617 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:37.532351    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr" event={"ID":"195909b1-e0bf-4679-84e3-50786cc7eaea","Type":"ContainerStarted","Data":"f8568961287ae2725915895a8347a54e94c47cc3e4bba476fb3f355ef5c291e8"}
Oct 31 23:17:37.532617 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:37.532472    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr" event={"ID":"195909b1-e0bf-4679-84e3-50786cc7eaea","Type":"ContainerStarted","Data":"3bf10819693fdbd41d12aa599ed252707774c868117c04fa39c9f2f9eb42575a"}
Oct 31 23:17:37.558859 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:37.555666    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:17:37.562789 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:37.562676    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr" podStartSLOduration=2.562655512 podStartE2EDuration="2.562655512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:17:37.560882062 +0000 UTC m=+1676.308071613" watchObservedRunningTime="2026-10-31 23:17:37.562655512 +0000 UTC m=+1676.309845020"
Oct 31 23:17:38.560248 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:38.560222    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:17:39.546259 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:39.546229    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:17:39.547168 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:39.547146    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:17:41.889988 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:41.889954    4301 scope.go:122] "RemoveContainer" containerID="71b6cea34e5e3c11f62f9d08160dbff715a8efa603dcddcc8c425e9aa5b1356b"
Oct 31 23:17:42.464909 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:42.464861    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:17:53.179496 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:53.179460    4301 patch_prober.go:29] interesting pod/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 23:17:53.179942 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:53.179533    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz" podUID="a3ef6611-0d56-41c3-9357-81e33a2fd67a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:17:54.994877 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:54.994799    4301 generic.go:356] "Generic (PLEG): container finished" podID="a3ef6611-0d56-41c3-9357-81e33a2fd67a" containerID="cb73ef2b0577265122157fc5171ab5630a7402bac27008f6b293da6d3ba371d3" exitCode=0
Oct 31 23:17:54.995407 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:54.994869    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz" event={"ID":"a3ef6611-0d56-41c3-9357-81e33a2fd67a","Type":"ContainerDied","Data":"cb73ef2b0577265122157fc5171ab5630a7402bac27008f6b293da6d3ba371d3"}
Oct 31 23:17:54.995495 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:54.995477    4301 scope.go:122] "RemoveContainer" containerID="cb73ef2b0577265122157fc5171ab5630a7402bac27008f6b293da6d3ba371d3"
Oct 31 23:17:54.997236 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:54.997205    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:17:54.997329 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:54.997270    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:17:54.997329 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:54.997306    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:17:56.003745 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:56.003713    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz" event={"ID":"a3ef6611-0d56-41c3-9357-81e33a2fd67a","Type":"ContainerStarted","Data":"803872df6429ac59bec602fc843243af9aea7bd08cde7d51fcfaca6473687042"}
Oct 31 23:17:56.589517 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:56.589482    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:17:56.589698 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:56.589618    4301 patch_prober.go:29] interesting pod/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 31 23:17:56.589698 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:56.589674    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz" podUID="a3ef6611-0d56-41c3-9357-81e33a2fd67a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 31 23:17:57.601523 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:57.601490    4301 patch_prober.go:29] interesting pod/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 31 23:17:57.602131 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:57.602097    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz" podUID="a3ef6611-0d56-41c3-9357-81e33a2fd67a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 31 23:17:58.590014 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:58.589982    4301 patch_prober.go:29] interesting pod/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 31 23:17:58.590204 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:58.590034    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz" podUID="a3ef6611-0d56-41c3-9357-81e33a2fd67a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 31 23:17:59.608391 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:59.591215    4301 patch_prober.go:29] interesting pod/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 31 23:17:59.609059 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:17:59.609022    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz" podUID="a3ef6611-0d56-41c3-9357-81e33a2fd67a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 31 23:18:00.589947 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:00.589913    4301 patch_prober.go:29] interesting pod/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 31 23:18:00.590128 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:00.589965    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz" podUID="a3ef6611-0d56-41c3-9357-81e33a2fd67a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 31 23:18:01.590180 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:01.590142    4301 patch_prober.go:29] interesting pod/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 31 23:18:01.590698 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:01.590203    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz" podUID="a3ef6611-0d56-41c3-9357-81e33a2fd67a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 31 23:18:02.603094 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:02.601171    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:18:02.603094 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:02.602931    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:18:02.605034 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:02.604968    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:18:02.657142 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:18:02.657100    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:18:17.353881 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:17.353848    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-85b90bac-bcdc55f5f-vtqx4"]
Oct 31 23:18:17.358607 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:17.358585    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-85b90bac-bcdc55f5f-vtqx4"
Oct 31 23:18:17.368550 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:17.368523    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-85b90bac\""
Oct 31 23:18:17.368838 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:17.368815    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-85b90bac-dockercfg-mfhg6\""
Oct 31 23:18:17.371391 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:17.371336    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-85b90bac-bcdc55f5f-vtqx4"]
Oct 31 23:18:17.435394 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:17.435347    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad66adfb-6f3e-43bf-8499-4b63285c1849-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-85b90bac-bcdc55f5f-vtqx4\" (UID: \"ad66adfb-6f3e-43bf-8499-4b63285c1849\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-85b90bac-bcdc55f5f-vtqx4"
Oct 31 23:18:17.435656 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:17.435632    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxpnk\" (UniqueName: \"kubernetes.io/projected/ad66adfb-6f3e-43bf-8499-4b63285c1849-kube-api-access-vxpnk\") pod \"payload-pre-processing-e2e-mt-85b90bac-bcdc55f5f-vtqx4\" (UID: \"ad66adfb-6f3e-43bf-8499-4b63285c1849\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-85b90bac-bcdc55f5f-vtqx4"
Oct 31 23:18:17.582382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:17.562649    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxpnk\" (UniqueName: \"kubernetes.io/projected/ad66adfb-6f3e-43bf-8499-4b63285c1849-kube-api-access-vxpnk\") pod \"payload-pre-processing-e2e-mt-85b90bac-bcdc55f5f-vtqx4\" (UID: \"ad66adfb-6f3e-43bf-8499-4b63285c1849\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-85b90bac-bcdc55f5f-vtqx4"
Oct 31 23:18:17.600388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:17.582898    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad66adfb-6f3e-43bf-8499-4b63285c1849-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-85b90bac-bcdc55f5f-vtqx4\" (UID: \"ad66adfb-6f3e-43bf-8499-4b63285c1849\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-85b90bac-bcdc55f5f-vtqx4"
Oct 31 23:18:17.600952 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:17.592330    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad66adfb-6f3e-43bf-8499-4b63285c1849-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-85b90bac-bcdc55f5f-vtqx4\" (UID: \"ad66adfb-6f3e-43bf-8499-4b63285c1849\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-85b90bac-bcdc55f5f-vtqx4"
Oct 31 23:18:17.641312 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:17.641279    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxpnk\" (UniqueName: \"kubernetes.io/projected/ad66adfb-6f3e-43bf-8499-4b63285c1849-kube-api-access-vxpnk\") pod \"payload-pre-processing-e2e-mt-85b90bac-bcdc55f5f-vtqx4\" (UID: \"ad66adfb-6f3e-43bf-8499-4b63285c1849\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-85b90bac-bcdc55f5f-vtqx4"
Oct 31 23:18:17.682681 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:17.682646    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-85b90bac-bcdc55f5f-vtqx4"
Oct 31 23:18:18.199213 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:18.199189    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-85b90bac-bcdc55f5f-vtqx4"]
Oct 31 23:18:18.200909 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:18:18.200874    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad66adfb_6f3e_43bf_8499_4b63285c1849.slice/crio-0e4c6b0b7d5de81381f8d033a93375ca9af81bfbd437d429b99799914624fd7e WatchSource:0}: Error finding container 0e4c6b0b7d5de81381f8d033a93375ca9af81bfbd437d429b99799914624fd7e: Status 404 returned error can't find the container with id 0e4c6b0b7d5de81381f8d033a93375ca9af81bfbd437d429b99799914624fd7e
Oct 31 23:18:18.670788 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:18.670740    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-85b90bac-bcdc55f5f-vtqx4" event={"ID":"ad66adfb-6f3e-43bf-8499-4b63285c1849","Type":"ContainerStarted","Data":"6f437b34b83e514b19c4a38abe51f1ceb7adbf74858311dc64c44f5a4646e08e"}
Oct 31 23:18:18.671388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:18.671344    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-85b90bac-bcdc55f5f-vtqx4"
Oct 31 23:18:18.671612 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:18.671592    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-85b90bac-bcdc55f5f-vtqx4" event={"ID":"ad66adfb-6f3e-43bf-8499-4b63285c1849","Type":"ContainerStarted","Data":"0e4c6b0b7d5de81381f8d033a93375ca9af81bfbd437d429b99799914624fd7e"}
Oct 31 23:18:19.545379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:19.542482    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:18:20.024540 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:20.024485    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:18:20.228530 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:20.228486    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:18:20.308593 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:20.308468    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 23:18:20.324590 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:20.324543    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:18:20.345515 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:20.345460    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:18:20.398902 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:20.398846    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97" podUID="c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:18:20.549549 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:20.549498    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:18:20.550090 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:20.549827    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 23:18:20.550605 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:20.550583    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:18:20.579434 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:20.571115    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-85b90bac-bcdc55f5f-vtqx4" podStartSLOduration=3.571095485 podStartE2EDuration="3.571095485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:18:18.71508875 +0000 UTC m=+1717.462278257" watchObservedRunningTime="2026-10-31 23:18:20.571095485 +0000 UTC m=+1719.318284990"
Oct 31 23:18:20.937413 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:20.916484    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:18:21.049382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:21.029269    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 23:18:21.062144 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:21.032460    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 31 23:18:21.095046 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:21.062401    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:18:21.095046 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:21.062532    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:18:21.095485 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:21.095453    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:18:21.233986 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:21.232781    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 23:18:21.288386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:21.238649    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:18:21.376021 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:21.375973    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:18:21.392197 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:21.385589    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:18:21.427580 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:21.427542    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:18:21.428695 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:21.428670    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:18:21.486433 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:21.481117    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:18:21.894383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:21.891674    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:18:22.055276 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:22.055229    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:18:22.057888 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:22.057855    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 23:18:22.058761 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:22.058733    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:18:22.113065 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:22.113022    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97" podUID="c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": context deadline exceeded"
Oct 31 23:18:22.235875 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:22.235777    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:18:22.383768 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:22.383725    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:18:22.424384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:22.391001    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:18:22.424384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:22.416121    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97" podUID="c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:18:22.429025 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:22.428997    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:18:22.430028 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:22.430005    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:18:22.430238 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:22.430225    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:18:22.432242 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:22.432212    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:18:22.486463 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:22.485599    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 23:18:22.486463 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:22.486423    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:18:22.520884 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:22.506704    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:18:22.961525 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:22.961480    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:18:23.059413 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:23.040656    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:18:23.063350 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:23.062289    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:18:23.100345 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:23.099792    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:18:23.211860 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:23.208280    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:18:23.274168 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:23.274048    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:18:23.362385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:23.361840    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:18:23.407829 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:23.407785    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:18:23.412947 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:23.412924    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:18:30.698014 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:30.697977    4301 generic.go:356] "Generic (PLEG): container finished" podID="195909b1-e0bf-4679-84e3-50786cc7eaea" containerID="f8568961287ae2725915895a8347a54e94c47cc3e4bba476fb3f355ef5c291e8" exitCode=0
Oct 31 23:18:30.698512 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:30.698038    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr" event={"ID":"195909b1-e0bf-4679-84e3-50786cc7eaea","Type":"ContainerDied","Data":"f8568961287ae2725915895a8347a54e94c47cc3e4bba476fb3f355ef5c291e8"}
Oct 31 23:18:30.698590 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:30.698575    4301 scope.go:122] "RemoveContainer" containerID="f8568961287ae2725915895a8347a54e94c47cc3e4bba476fb3f355ef5c291e8"
Oct 31 23:18:30.700728 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:30.700697    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:18:30.700829 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:30.700780    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:18:30.700829 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:30.700816    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:18:30.856108 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:30.856078    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-85b90bac-bcdc55f5f-vtqx4"
Oct 31 23:18:31.760760 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:31.759662    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr" event={"ID":"195909b1-e0bf-4679-84e3-50786cc7eaea","Type":"ContainerStarted","Data":"2a1750e13aa82eed80f46e0007e854c4b42f473c1b9ac4faaa1720815e3015d4"}
Oct 31 23:18:32.555385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:32.554853    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:18:32.555385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:32.554990    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 31 23:18:32.555385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:32.555033    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr" podUID="195909b1-e0bf-4679-84e3-50786cc7eaea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 31 23:18:32.575008 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:18:32.574978    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:18:33.555892 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:33.555851    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 31 23:18:33.556469 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:33.555935    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr" podUID="195909b1-e0bf-4679-84e3-50786cc7eaea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 31 23:18:34.569408 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:34.565776    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 31 23:18:34.569408 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:34.565854    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr" podUID="195909b1-e0bf-4679-84e3-50786cc7eaea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 31 23:18:35.023379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:35.023326    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8e05f76-jw5n4"]
Oct 31 23:18:35.026948 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:35.026922    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8e05f76-jw5n4"]
Oct 31 23:18:35.424144 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:35.424104    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbf2a79d-5e58-490e-9947-68226f151f1f" path="/var/lib/kubelet/pods/dbf2a79d-5e58-490e-9947-68226f151f1f/volumes"
Oct 31 23:18:35.555768 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:35.555729    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 31 23:18:35.555941 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:35.555811    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr" podUID="195909b1-e0bf-4679-84e3-50786cc7eaea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 31 23:18:36.565386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:36.562544    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 31 23:18:36.565386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:36.562623    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr" podUID="195909b1-e0bf-4679-84e3-50786cc7eaea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 31 23:18:37.559771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:37.559741    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:18:37.560224 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:37.560205    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:18:37.560872 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:37.560854    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:18:39.642803 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:39.642758    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"]
Oct 31 23:18:39.648337 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:39.648310    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:18:39.668696 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:39.668667    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"]
Oct 31 23:18:39.679841 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:39.679815    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-270e8c-openshift-default-dockercfg-sbdqn\""
Oct 31 23:18:39.834385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:39.817966    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnb5n\" (UniqueName: \"kubernetes.io/projected/394e1575-7171-45a2-8d19-ef551eaad0ea-kube-api-access-qnb5n\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:18:39.834698 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:39.834666    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/394e1575-7171-45a2-8d19-ef551eaad0ea-istiod-ca-cert\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:18:39.841382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:39.834893    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-istio-envoy\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:18:39.841871 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:39.841836    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-workload-certs\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:18:39.842160 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:39.842138    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-credential-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:18:39.859571 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:39.859537    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-workload-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:18:39.859869 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:39.859848    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/394e1575-7171-45a2-8d19-ef551eaad0ea-istio-token\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:18:39.860125 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:39.860109    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-istio-data\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:18:39.860278 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:39.860263    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/394e1575-7171-45a2-8d19-ef551eaad0ea-istio-podinfo\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:18:39.965463 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:39.964832    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-istio-envoy\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:18:39.965463 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:39.964934    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-workload-certs\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:18:39.965463 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:39.964983    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-credential-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:18:39.965463 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:39.965025    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-workload-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:18:39.965463 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:39.965093    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/394e1575-7171-45a2-8d19-ef551eaad0ea-istio-token\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:18:39.965463 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:39.965220    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-istio-data\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:18:39.965463 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:39.965278    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/394e1575-7171-45a2-8d19-ef551eaad0ea-istio-podinfo\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:18:39.965463 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:39.965417    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnb5n\" (UniqueName: \"kubernetes.io/projected/394e1575-7171-45a2-8d19-ef551eaad0ea-kube-api-access-qnb5n\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:18:39.965463 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:39.965455    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/394e1575-7171-45a2-8d19-ef551eaad0ea-istiod-ca-cert\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:18:39.969381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:39.966299    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/394e1575-7171-45a2-8d19-ef551eaad0ea-istiod-ca-cert\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:18:39.985378 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:39.980579    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-workload-certs\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:18:39.985378 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:39.980847    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-credential-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:18:39.985378 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:39.981097    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-workload-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:18:39.985378 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:39.982152    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-istio-data\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:18:39.997318 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:39.997250    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/394e1575-7171-45a2-8d19-ef551eaad0ea-istio-podinfo\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:18:40.006378 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:40.003097    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/394e1575-7171-45a2-8d19-ef551eaad0ea-istio-token\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:18:40.006807 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:40.006779    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-istio-envoy\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:18:40.022602 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:40.022564    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnb5n\" (UniqueName: \"kubernetes.io/projected/394e1575-7171-45a2-8d19-ef551eaad0ea-kube-api-access-qnb5n\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:18:40.285860 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:40.285745    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:18:40.679478 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:40.679453    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"]
Oct 31 23:18:40.681092 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:18:40.681061    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod394e1575_7171_45a2_8d19_ef551eaad0ea.slice/crio-d7932243ce9c51138cde57a996faf8fc01aa06e71558fcd729f1ec4bdfba07d4 WatchSource:0}: Error finding container d7932243ce9c51138cde57a996faf8fc01aa06e71558fcd729f1ec4bdfba07d4: Status 404 returned error can't find the container with id d7932243ce9c51138cde57a996faf8fc01aa06e71558fcd729f1ec4bdfba07d4
Oct 31 23:18:40.683727 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:40.683691    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:18:40.683828 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:40.683772    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:18:40.683935 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:40.683826    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:18:40.920862 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:40.920825    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p" event={"ID":"394e1575-7171-45a2-8d19-ef551eaad0ea","Type":"ContainerStarted","Data":"0f85ac85e722e495f38549d8df0595653b60c874b0a2047a4ac9e9aed8bf6864"}
Oct 31 23:18:40.921043 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:40.920889    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p" event={"ID":"394e1575-7171-45a2-8d19-ef551eaad0ea","Type":"ContainerStarted","Data":"d7932243ce9c51138cde57a996faf8fc01aa06e71558fcd729f1ec4bdfba07d4"}
Oct 31 23:18:40.946515 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:40.946390    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p" podStartSLOduration=1.9463710760000001 podStartE2EDuration="1.946371076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:18:40.941720222 +0000 UTC m=+1739.688909740" watchObservedRunningTime="2026-10-31 23:18:40.946371076 +0000 UTC m=+1739.693560580"
Oct 31 23:18:41.287407 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:41.287299    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:18:41.294453 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:41.294423    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:18:41.989499 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:41.989459    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:18:42.017409 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:42.014656    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:18:44.262022 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:44.261986    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:18:45.086380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:45.067307    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-270e8c-cfb487cd-xpmx2"]
Oct 31 23:18:45.165388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:45.150320    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-270e8c-cfb487cd-xpmx2"
Oct 31 23:18:45.249401 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:45.239111    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-270e8c-dockercfg-msbww\""
Oct 31 23:18:45.249401 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:45.239310    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-270e8c\""
Oct 31 23:18:45.285382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:45.281231    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-270e8c-cfb487cd-xpmx2"]
Oct 31 23:18:45.301382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:45.290641    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95sds\" (UniqueName: \"kubernetes.io/projected/59d49565-55d0-4f45-8dd4-64bbe81bad3d-kube-api-access-95sds\") pod \"payload-processing-e2e-shared-b-w3-270e8c-cfb487cd-xpmx2\" (UID: \"59d49565-55d0-4f45-8dd4-64bbe81bad3d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-270e8c-cfb487cd-xpmx2"
Oct 31 23:18:45.301382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:45.290910    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59d49565-55d0-4f45-8dd4-64bbe81bad3d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-270e8c-cfb487cd-xpmx2\" (UID: \"59d49565-55d0-4f45-8dd4-64bbe81bad3d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-270e8c-cfb487cd-xpmx2"
Oct 31 23:18:45.418456 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:45.418404    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59d49565-55d0-4f45-8dd4-64bbe81bad3d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-270e8c-cfb487cd-xpmx2\" (UID: \"59d49565-55d0-4f45-8dd4-64bbe81bad3d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-270e8c-cfb487cd-xpmx2"
Oct 31 23:18:45.418991 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:45.418962    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95sds\" (UniqueName: \"kubernetes.io/projected/59d49565-55d0-4f45-8dd4-64bbe81bad3d-kube-api-access-95sds\") pod \"payload-processing-e2e-shared-b-w3-270e8c-cfb487cd-xpmx2\" (UID: \"59d49565-55d0-4f45-8dd4-64bbe81bad3d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-270e8c-cfb487cd-xpmx2"
Oct 31 23:18:45.420694 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:45.420653    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59d49565-55d0-4f45-8dd4-64bbe81bad3d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-270e8c-cfb487cd-xpmx2\" (UID: \"59d49565-55d0-4f45-8dd4-64bbe81bad3d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-270e8c-cfb487cd-xpmx2"
Oct 31 23:18:45.489088 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:45.479592    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95sds\" (UniqueName: \"kubernetes.io/projected/59d49565-55d0-4f45-8dd4-64bbe81bad3d-kube-api-access-95sds\") pod \"payload-processing-e2e-shared-b-w3-270e8c-cfb487cd-xpmx2\" (UID: \"59d49565-55d0-4f45-8dd4-64bbe81bad3d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-270e8c-cfb487cd-xpmx2"
Oct 31 23:18:45.702031 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:45.699650    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-270e8c-cfb487cd-xpmx2"
Oct 31 23:18:46.516387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:46.470867    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-270e8c-cfb487cd-xpmx2"]
Oct 31 23:18:47.368453 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:47.367839    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-270e8c-cfb487cd-xpmx2" event={"ID":"59d49565-55d0-4f45-8dd4-64bbe81bad3d","Type":"ContainerStarted","Data":"a31acc4094c020a340ea7a6722a401edb838c6494e8642150f801c096297791b"}
Oct 31 23:18:47.368453 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:47.367904    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-270e8c-cfb487cd-xpmx2" event={"ID":"59d49565-55d0-4f45-8dd4-64bbe81bad3d","Type":"ContainerStarted","Data":"6d77397aa88a4e3e928b4debb4449ad9b44cca457ab8ee869d8203913d305e86"}
Oct 31 23:18:47.379096 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:47.369036    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-270e8c-cfb487cd-xpmx2"
Oct 31 23:18:47.397419 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:47.397166    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-270e8c-cfb487cd-xpmx2" podStartSLOduration=2.397149136 podStartE2EDuration="2.397149136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:18:47.395630863 +0000 UTC m=+1746.142820369" watchObservedRunningTime="2026-10-31 23:18:47.397149136 +0000 UTC m=+1746.144338643"
Oct 31 23:18:59.416944 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:18:59.416861    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-270e8c-cfb487cd-xpmx2"
Oct 31 23:19:02.546734 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:19:02.546699    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:19:03.058591 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:03.058556    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe640623-tvcgh"]
Oct 31 23:19:03.079219 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:03.079180    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe640623-tvcgh"]
Oct 31 23:19:03.406879 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:03.406847    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13bf0fc3-47ae-4bb3-8326-6a3e059d47da" path="/var/lib/kubelet/pods/13bf0fc3-47ae-4bb3-8326-6a3e059d47da/volumes"
Oct 31 23:19:20.375476 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:20.375437    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"]
Oct 31 23:19:20.382926 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:20.382886    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"]
Oct 31 23:19:20.383100 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:20.382987    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:19:20.386315 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:20.386283    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-4528ed-openshift-default-dockercfg-sdddc\""
Oct 31 23:19:20.505384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:20.498464    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a97819-25d1-43c8-b104-ee21999c5f68-istio-token\") pod \"e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") " pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:19:20.518819 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:20.518776    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-istio-data\") pod \"e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") " pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:19:20.519256 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:20.519216    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a97819-25d1-43c8-b104-ee21999c5f68-istio-podinfo\") pod \"e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") " pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:19:20.519622 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:20.519589    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-istio-envoy\") pod \"e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") " pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:19:20.536053 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:20.523471    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-workload-socket\") pod \"e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") " pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:19:20.536053 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:20.523505    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a97819-25d1-43c8-b104-ee21999c5f68-istiod-ca-cert\") pod \"e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") " pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:19:20.536053 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:20.523597    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-credential-socket\") pod \"e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") " pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:19:20.536053 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:20.523703    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-459hz\" (UniqueName: \"kubernetes.io/projected/74a97819-25d1-43c8-b104-ee21999c5f68-kube-api-access-459hz\") pod \"e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") " pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:19:20.536053 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:20.523789    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-workload-certs\") pod \"e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") " pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:19:20.642481 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:20.640530    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-workload-certs\") pod \"e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") " pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:19:20.642481 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:20.640642    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a97819-25d1-43c8-b104-ee21999c5f68-istio-token\") pod \"e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") " pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:19:20.642481 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:20.640712    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-istio-data\") pod \"e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") " pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:19:20.642481 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:20.640766    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a97819-25d1-43c8-b104-ee21999c5f68-istio-podinfo\") pod \"e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") " pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:19:20.642481 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:20.640903    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-istio-envoy\") pod \"e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") " pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:19:20.642481 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:20.640977    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-workload-socket\") pod \"e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") " pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:19:20.642481 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:20.641001    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a97819-25d1-43c8-b104-ee21999c5f68-istiod-ca-cert\") pod \"e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") " pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:19:20.642481 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:20.641098    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-credential-socket\") pod \"e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") " pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:19:20.642481 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:20.641195    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-459hz\" (UniqueName: \"kubernetes.io/projected/74a97819-25d1-43c8-b104-ee21999c5f68-kube-api-access-459hz\") pod \"e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") " pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:19:20.642481 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:20.642106    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-workload-certs\") pod \"e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") " pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:19:20.643150 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:20.642661    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-istio-data\") pod \"e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") " pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:19:20.652417 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:20.644245    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-workload-socket\") pod \"e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") " pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:19:20.652417 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:20.646183    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a97819-25d1-43c8-b104-ee21999c5f68-istiod-ca-cert\") pod \"e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") " pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:19:20.692443 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:20.684331    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-credential-socket\") pod \"e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") " pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:19:20.703380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:20.696714    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-istio-envoy\") pod \"e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") " pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:19:20.711391 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:20.704283    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a97819-25d1-43c8-b104-ee21999c5f68-istio-podinfo\") pod \"e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") " pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:19:20.711391 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:20.705167    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-459hz\" (UniqueName: \"kubernetes.io/projected/74a97819-25d1-43c8-b104-ee21999c5f68-kube-api-access-459hz\") pod \"e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") " pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:19:20.731393 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:20.724250    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a97819-25d1-43c8-b104-ee21999c5f68-istio-token\") pod \"e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") " pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:19:21.001522 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:21.001420    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:19:21.316149 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:21.316066    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"]
Oct 31 23:19:21.321061 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:19:21.321032    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a97819_25d1_43c8_b104_ee21999c5f68.slice/crio-ae0eb8e55f3e7779a6b2b20e4ef3bf3863a4f0911774f770a477fc8b20bb566f WatchSource:0}: Error finding container ae0eb8e55f3e7779a6b2b20e4ef3bf3863a4f0911774f770a477fc8b20bb566f: Status 404 returned error can't find the container with id ae0eb8e55f3e7779a6b2b20e4ef3bf3863a4f0911774f770a477fc8b20bb566f
Oct 31 23:19:21.324563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:21.324525    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:19:21.324666 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:21.324598    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:19:21.324666 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:21.324630    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:19:21.984407 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:21.984242    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x" event={"ID":"74a97819-25d1-43c8-b104-ee21999c5f68","Type":"ContainerStarted","Data":"9457e388751789f92d5088d58f4cd2dda892459c3ddd36288972da7dea82957f"}
Oct 31 23:19:21.984407 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:21.984411    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x" event={"ID":"74a97819-25d1-43c8-b104-ee21999c5f68","Type":"ContainerStarted","Data":"ae0eb8e55f3e7779a6b2b20e4ef3bf3863a4f0911774f770a477fc8b20bb566f"}
Oct 31 23:19:22.002025 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:22.001995    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:19:22.014521 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:22.014468    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x" podStartSLOduration=2.014453592 podStartE2EDuration="2.014453592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:19:22.007501969 +0000 UTC m=+1780.754691486" watchObservedRunningTime="2026-10-31 23:19:22.014453592 +0000 UTC m=+1780.761643096"
Oct 31 23:19:22.019033 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:22.019007    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:19:22.991096 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:22.991067    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:19:22.992047 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:22.992029    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:19:26.176695 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:26.176661    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"]
Oct 31 23:19:26.215885 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:26.215821    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"]
Oct 31 23:19:26.216192 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:26.216166    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:19:26.291380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:26.290703    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-479107-openshift-default-dockercfg-pnn5x\""
Oct 31 23:19:26.469673 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:26.465931    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-istio-data\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:19:26.469673 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:26.466047    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hshl\" (UniqueName: \"kubernetes.io/projected/f7560a60-cbba-43ba-81a1-946c57663b80-kube-api-access-8hshl\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:19:26.469673 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:26.466091    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-istio-envoy\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:19:26.469673 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:26.466212    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7560a60-cbba-43ba-81a1-946c57663b80-istiod-ca-cert\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:19:26.469673 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:26.466249    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-credential-socket\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:19:26.469673 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:26.466377    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-workload-socket\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:19:26.469673 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:26.466433    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7560a60-cbba-43ba-81a1-946c57663b80-istio-token\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:19:26.469673 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:26.466485    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-workload-certs\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:19:26.469673 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:26.466576    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7560a60-cbba-43ba-81a1-946c57663b80-istio-podinfo\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:19:26.573381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:26.569667    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7560a60-cbba-43ba-81a1-946c57663b80-istio-token\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:19:26.573381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:26.570072    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-workload-certs\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:19:26.573381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:26.570147    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7560a60-cbba-43ba-81a1-946c57663b80-istio-podinfo\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:19:26.573381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:26.570239    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-istio-data\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:19:26.573381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:26.570333    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hshl\" (UniqueName: \"kubernetes.io/projected/f7560a60-cbba-43ba-81a1-946c57663b80-kube-api-access-8hshl\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:19:26.573381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:26.570392    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-istio-envoy\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:19:26.573381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:26.570520    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7560a60-cbba-43ba-81a1-946c57663b80-istiod-ca-cert\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:19:26.573381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:26.570589    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-credential-socket\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:19:26.573381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:26.570684    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-workload-socket\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:19:26.573381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:26.571651    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-workload-certs\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:19:26.573381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:26.572117    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-workload-socket\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:19:26.573381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:26.572735    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7560a60-cbba-43ba-81a1-946c57663b80-istiod-ca-cert\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:19:26.574146 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:26.573829    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-credential-socket\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:19:26.574146 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:26.573858    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-istio-data\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:19:26.574607 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:26.574559    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"]
Oct 31 23:19:26.576926 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:26.576903    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-istio-envoy\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:19:26.594387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:26.583453    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7560a60-cbba-43ba-81a1-946c57663b80-istio-podinfo\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:19:26.594387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:26.584747    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hshl\" (UniqueName: \"kubernetes.io/projected/f7560a60-cbba-43ba-81a1-946c57663b80-kube-api-access-8hshl\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:19:26.594387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:26.585280    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7560a60-cbba-43ba-81a1-946c57663b80-istio-token\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:19:26.837868 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:26.837793    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:19:27.307792 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:27.307763    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"]
Oct 31 23:19:27.311332 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:19:27.311305    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7560a60_cbba_43ba_81a1_946c57663b80.slice/crio-37dfcc5efb26c796f7a89abb1d5a6e151608f42af2988151a6476650531583e5 WatchSource:0}: Error finding container 37dfcc5efb26c796f7a89abb1d5a6e151608f42af2988151a6476650531583e5: Status 404 returned error can't find the container with id 37dfcc5efb26c796f7a89abb1d5a6e151608f42af2988151a6476650531583e5
Oct 31 23:19:27.314489 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:27.314455    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:19:27.314601 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:27.314525    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:19:27.314601 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:27.314555    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:19:28.085123 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:28.085083    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" event={"ID":"f7560a60-cbba-43ba-81a1-946c57663b80","Type":"ContainerStarted","Data":"b711c656e8708f7a548a7453b46176951c1341be574dfd1c54c987141351ef13"}
Oct 31 23:19:28.085305 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:28.085133    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" event={"ID":"f7560a60-cbba-43ba-81a1-946c57663b80","Type":"ContainerStarted","Data":"37dfcc5efb26c796f7a89abb1d5a6e151608f42af2988151a6476650531583e5"}
Oct 31 23:19:28.109373 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:28.109307    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podStartSLOduration=2.109287949 podStartE2EDuration="2.109287949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:19:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:19:28.106593194 +0000 UTC m=+1786.853782699" watchObservedRunningTime="2026-10-31 23:19:28.109287949 +0000 UTC m=+1786.856477449"
Oct 31 23:19:28.838124 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:28.838091    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:19:28.839543 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:28.839520    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:19:28.839655 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:28.839561    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:19:29.846695 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:29.840772    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:19:29.846695 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:29.840832    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:19:30.838436 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:30.838401    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:19:30.838708 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:30.838456    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:19:31.838875 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:31.838830    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:19:31.839313 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:31.838894    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:19:32.584951 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:19:32.584915    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:19:32.839132 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:32.839052    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:19:32.839132 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:32.839106    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:19:33.838348 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:33.838310    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:19:33.838567 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:33.838390    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:19:34.855385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:34.853438    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:19:34.855385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:34.853527    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:19:35.838370 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:35.838317    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:19:35.838584 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:35.838419    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:19:36.841387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:36.840679    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:19:36.841387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:36.840745    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:19:37.838528 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:37.838491    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:19:37.838711 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:37.838552    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:19:38.840095 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:38.840057    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:19:38.840806 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:38.840764    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:19:39.839031 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:39.838991    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:19:39.839274 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:39.839056    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:19:40.838839 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:40.838788    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:19:40.839244 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:40.838840    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:19:41.714959 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:41.714927    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s9nl_1f0042ee-6258-4aa8-af48-7568f2f2e8a1/ovn-acl-logging/0.log"
Oct 31 23:19:41.719026 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:41.718995    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s9nl_1f0042ee-6258-4aa8-af48-7568f2f2e8a1/ovn-acl-logging/0.log"
Oct 31 23:19:41.838121 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:41.838088    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:19:41.838403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:41.838351    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:19:41.838509 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:41.838434    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:19:42.067008 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:42.066911    4301 scope.go:122] "RemoveContainer" containerID="215edbd41a4bd8e4963941427e1de3418a0499f65810ce15eefaa1a9a6e7bb42"
Oct 31 23:19:42.110806 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:42.110745    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main" containerID="cri-o://91e981e471f18cd72cb7a666fce8f21853dbfc76270ed7ad1638c86466cb815f" gracePeriod=45
Oct 31 23:19:42.312992 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:42.312950    4301 generic.go:356] "Generic (PLEG): container finished" podID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerID="91e981e471f18cd72cb7a666fce8f21853dbfc76270ed7ad1638c86466cb815f" exitCode=0
Oct 31 23:19:42.313877 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:42.313841    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" event={"ID":"ff4f6df9-e1f0-4163-895c-7df6b065b033","Type":"ContainerDied","Data":"91e981e471f18cd72cb7a666fce8f21853dbfc76270ed7ad1638c86466cb815f"}
Oct 31 23:19:42.510626 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:42.510586    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:19:42.579164 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:42.579128    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff4f6df9-e1f0-4163-895c-7df6b065b033-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff4f6df9-e1f0-4163-895c-7df6b065b033-model-cache\") pod \"ff4f6df9-e1f0-4163-895c-7df6b065b033\" (UID: \"ff4f6df9-e1f0-4163-895c-7df6b065b033\") "
Oct 31 23:19:42.579353 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:42.579192    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff4f6df9-e1f0-4163-895c-7df6b065b033-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff4f6df9-e1f0-4163-895c-7df6b065b033-tmp-dir\") pod \"ff4f6df9-e1f0-4163-895c-7df6b065b033\" (UID: \"ff4f6df9-e1f0-4163-895c-7df6b065b033\") "
Oct 31 23:19:42.579496 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:42.579466    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff4f6df9-e1f0-4163-895c-7df6b065b033-model-cache" pod "ff4f6df9-e1f0-4163-895c-7df6b065b033" (UID: "ff4f6df9-e1f0-4163-895c-7df6b065b033"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:19:42.579576 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:42.579495    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff4f6df9-e1f0-4163-895c-7df6b065b033-tls-certs\" (UniqueName: \"kubernetes.io/secret/ff4f6df9-e1f0-4163-895c-7df6b065b033-tls-certs\") pod \"ff4f6df9-e1f0-4163-895c-7df6b065b033\" (UID: \"ff4f6df9-e1f0-4163-895c-7df6b065b033\") "
Oct 31 23:19:42.579576 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:42.579545    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff4f6df9-e1f0-4163-895c-7df6b065b033-kube-api-access-j4qgw\" (UniqueName: \"kubernetes.io/projected/ff4f6df9-e1f0-4163-895c-7df6b065b033-kube-api-access-j4qgw\") pod \"ff4f6df9-e1f0-4163-895c-7df6b065b033\" (UID: \"ff4f6df9-e1f0-4163-895c-7df6b065b033\") "
Oct 31 23:19:42.579576 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:42.579563    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff4f6df9-e1f0-4163-895c-7df6b065b033-tmp-dir" pod "ff4f6df9-e1f0-4163-895c-7df6b065b033" (UID: "ff4f6df9-e1f0-4163-895c-7df6b065b033"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:19:42.579739 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:42.579587    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff4f6df9-e1f0-4163-895c-7df6b065b033-home\" (UniqueName: \"kubernetes.io/empty-dir/ff4f6df9-e1f0-4163-895c-7df6b065b033-home\") pod \"ff4f6df9-e1f0-4163-895c-7df6b065b033\" (UID: \"ff4f6df9-e1f0-4163-895c-7df6b065b033\") "
Oct 31 23:19:42.579739 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:42.579624    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff4f6df9-e1f0-4163-895c-7df6b065b033-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff4f6df9-e1f0-4163-895c-7df6b065b033-dshm\") pod \"ff4f6df9-e1f0-4163-895c-7df6b065b033\" (UID: \"ff4f6df9-e1f0-4163-895c-7df6b065b033\") "
Oct 31 23:19:42.580546 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:42.579947    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff4f6df9-e1f0-4163-895c-7df6b065b033-home" pod "ff4f6df9-e1f0-4163-895c-7df6b065b033" (UID: "ff4f6df9-e1f0-4163-895c-7df6b065b033"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:19:42.580895 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:42.580833    4301 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff4f6df9-e1f0-4163-895c-7df6b065b033-tmp-dir\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:19:42.580895 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:42.580860    4301 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff4f6df9-e1f0-4163-895c-7df6b065b033-home\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:19:42.580895 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:42.580874    4301 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff4f6df9-e1f0-4163-895c-7df6b065b033-model-cache\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:19:42.582337 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:42.582306    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff4f6df9-e1f0-4163-895c-7df6b065b033-tls-certs" pod "ff4f6df9-e1f0-4163-895c-7df6b065b033" (UID: "ff4f6df9-e1f0-4163-895c-7df6b065b033"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:19:42.582482 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:42.582453    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff4f6df9-e1f0-4163-895c-7df6b065b033-dshm" pod "ff4f6df9-e1f0-4163-895c-7df6b065b033" (UID: "ff4f6df9-e1f0-4163-895c-7df6b065b033"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:19:42.582617 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:42.582596    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff4f6df9-e1f0-4163-895c-7df6b065b033-kube-api-access-j4qgw" pod "ff4f6df9-e1f0-4163-895c-7df6b065b033" (UID: "ff4f6df9-e1f0-4163-895c-7df6b065b033"). InnerVolumeSpecName "kube-api-access-j4qgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:19:42.682523 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:42.682437    4301 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff4f6df9-e1f0-4163-895c-7df6b065b033-tls-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:19:42.682523 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:42.682469    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4qgw\" (UniqueName: \"kubernetes.io/projected/ff4f6df9-e1f0-4163-895c-7df6b065b033-kube-api-access-j4qgw\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:19:42.682523 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:42.682479    4301 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff4f6df9-e1f0-4163-895c-7df6b065b033-dshm\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:19:42.838782 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:42.838738    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:19:42.870390 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:42.863440    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:19:43.322234 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:43.322206    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"
Oct 31 23:19:43.322234 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:43.322216    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4" event={"ID":"ff4f6df9-e1f0-4163-895c-7df6b065b033","Type":"ContainerDied","Data":"d3ecfce430072ffa0e846071a01be8c2bcb2df1b00a1e062c981ed1fc228af57"}
Oct 31 23:19:43.322774 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:43.322257    4301 scope.go:122] "RemoveContainer" containerID="91e981e471f18cd72cb7a666fce8f21853dbfc76270ed7ad1638c86466cb815f"
Oct 31 23:19:43.347104 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:43.347077    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"]
Oct 31 23:19:43.350849 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:43.350826    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a16dfb/premium-w0-a16dfb-kserve-64c9b45554-bckr4"]
Oct 31 23:19:43.406924 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:43.406893    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" path="/var/lib/kubelet/pods/ff4f6df9-e1f0-4163-895c-7df6b065b033/volumes"
Oct 31 23:19:43.838511 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:43.838476    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:19:43.838699 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:43.838524    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:19:44.838304 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:44.838270    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:19:44.838721 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:44.838332    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:19:45.838655 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:45.838618    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:19:45.839049 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:45.838685    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:19:46.840979 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:46.840890    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:19:46.841918 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:46.841576    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:19:47.839077 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:47.839043    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:19:47.839262 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:47.839106    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:19:48.838696 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:48.838661    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:19:48.839098 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:48.838722    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:19:49.838573 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:49.838534    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:19:49.838778 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:49.838601    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:19:50.839130 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:50.839094    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:19:50.839544 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:50.839146    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:19:51.839099 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:51.839061    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:19:51.839283 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:51.839132    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:19:52.838807 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:52.838768    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:19:52.839032 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:52.838850    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:19:53.838389 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:53.838326    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:19:53.838778 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:53.838427    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:19:54.844956 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:54.844919    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:19:54.846962 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:54.846901    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:19:55.748414 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:55.748337    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"]
Oct 31 23:19:55.748619 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:55.748595    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz" podUID="13c9f33c-af79-4938-b54f-142dc2f72218" containerName="istio-proxy" containerID="cri-o://1a4188deb86605e67944e8210642d0dd1d1d233c923ccf29c897f6a81d5f1066" gracePeriod=30
Oct 31 23:19:55.752671 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:55.752644    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"]
Oct 31 23:19:55.752894 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:55.752872    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz" podUID="3b0d47aa-36ab-46fa-b41f-20a71f2256cd" containerName="istio-proxy" containerID="cri-o://8b7f17202fa4b84b3448c8fc2b9dff61a7e2c111c38a19277ce96ba1bea1125d" gracePeriod=30
Oct 31 23:19:55.838494 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:55.838459    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:19:55.838693 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:55.838521    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:19:56.838828 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:56.838788    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:19:56.839267 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:56.838856    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:19:57.838251 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:57.838214    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:19:57.838472 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:57.838273    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:19:57.838472 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:57.838314    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:19:57.839036 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:57.839003    4301 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="istio-proxy" containerStatusID={"Type":"cri-o","ID":"b711c656e8708f7a548a7453b46176951c1341be574dfd1c54c987141351ef13"} pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" containerMessage="Container istio-proxy failed startup probe, will be restarted"
Oct 31 23:19:57.839470 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:19:57.839067    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" containerID="cri-o://b711c656e8708f7a548a7453b46176951c1341be574dfd1c54c987141351ef13" gracePeriod=30
Oct 31 23:20:01.045793 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.045726    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:20:01.117077 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.117047    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istio-token\" (UniqueName: \"kubernetes.io/projected/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istio-token\") pod \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") "
Oct 31 23:20:01.117260 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.117101    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-credential-socket\") pod \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") "
Oct 31 23:20:01.117260 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.117211    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istiod-ca-cert\") pod \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") "
Oct 31 23:20:01.117417 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.117298    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istio-envoy\") pod \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") "
Oct 31 23:20:01.117417 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.117398    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-kube-api-access-q72qt\" (UniqueName: \"kubernetes.io/projected/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-kube-api-access-q72qt\") pod \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") "
Oct 31 23:20:01.117417 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.117400    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-credential-socket" pod "3b0d47aa-36ab-46fa-b41f-20a71f2256cd" (UID: "3b0d47aa-36ab-46fa-b41f-20a71f2256cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:20:01.117572 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.117428    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-workload-certs\") pod \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") "
Oct 31 23:20:01.117694 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.117675    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-workload-certs" pod "3b0d47aa-36ab-46fa-b41f-20a71f2256cd" (UID: "3b0d47aa-36ab-46fa-b41f-20a71f2256cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:20:01.117760 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.117698    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istiod-ca-cert" pod "3b0d47aa-36ab-46fa-b41f-20a71f2256cd" (UID: "3b0d47aa-36ab-46fa-b41f-20a71f2256cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:20:01.117813 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.117798    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istio-podinfo\") pod \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") "
Oct 31 23:20:01.117866 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.117833    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-workload-socket\") pod \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") "
Oct 31 23:20:01.117919 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.117862    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istio-data\") pod \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\" (UID: \"3b0d47aa-36ab-46fa-b41f-20a71f2256cd\") "
Oct 31 23:20:01.118133 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.118110    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-workload-socket" pod "3b0d47aa-36ab-46fa-b41f-20a71f2256cd" (UID: "3b0d47aa-36ab-46fa-b41f-20a71f2256cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:20:01.118312 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.118291    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istio-data" pod "3b0d47aa-36ab-46fa-b41f-20a71f2256cd" (UID: "3b0d47aa-36ab-46fa-b41f-20a71f2256cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:20:01.118929 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.118905    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:20:01.119037 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.118934    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:20:01.119037 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.118949    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:20:01.119037 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.118962    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:20:01.119037 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.118975    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:20:01.119914 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.119888    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-kube-api-access-q72qt" pod "3b0d47aa-36ab-46fa-b41f-20a71f2256cd" (UID: "3b0d47aa-36ab-46fa-b41f-20a71f2256cd"). InnerVolumeSpecName "kube-api-access-q72qt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:20:01.120025 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.119914    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istio-token" pod "3b0d47aa-36ab-46fa-b41f-20a71f2256cd" (UID: "3b0d47aa-36ab-46fa-b41f-20a71f2256cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:20:01.120280 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.120256    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istio-envoy" pod "3b0d47aa-36ab-46fa-b41f-20a71f2256cd" (UID: "3b0d47aa-36ab-46fa-b41f-20a71f2256cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:20:01.122510 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.122486    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istio-podinfo" pod "3b0d47aa-36ab-46fa-b41f-20a71f2256cd" (UID: "3b0d47aa-36ab-46fa-b41f-20a71f2256cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:20:01.220250 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.220214    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:20:01.220250 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.220244    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q72qt\" (UniqueName: \"kubernetes.io/projected/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-kube-api-access-q72qt\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:20:01.220250 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.220255    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:20:01.220535 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.220263    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0d47aa-36ab-46fa-b41f-20a71f2256cd-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:20:01.535814 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.535789    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:20:01.554864 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.554835    4301 generic.go:356] "Generic (PLEG): container finished" podID="13c9f33c-af79-4938-b54f-142dc2f72218" containerID="1a4188deb86605e67944e8210642d0dd1d1d233c923ccf29c897f6a81d5f1066" exitCode=0
Oct 31 23:20:01.555009 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.554886    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"
Oct 31 23:20:01.555009 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.554919    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz" event={"ID":"13c9f33c-af79-4938-b54f-142dc2f72218","Type":"ContainerDied","Data":"1a4188deb86605e67944e8210642d0dd1d1d233c923ccf29c897f6a81d5f1066"}
Oct 31 23:20:01.555009 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.554977    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz" event={"ID":"13c9f33c-af79-4938-b54f-142dc2f72218","Type":"ContainerDied","Data":"a211e6d6768d35751c97187f528ed99230b55b66032c59552e40e0586d414756"}
Oct 31 23:20:01.555009 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.555004    4301 scope.go:122] "RemoveContainer" containerID="1a4188deb86605e67944e8210642d0dd1d1d233c923ccf29c897f6a81d5f1066"
Oct 31 23:20:01.556849 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.556815    4301 generic.go:356] "Generic (PLEG): container finished" podID="3b0d47aa-36ab-46fa-b41f-20a71f2256cd" containerID="8b7f17202fa4b84b3448c8fc2b9dff61a7e2c111c38a19277ce96ba1bea1125d" exitCode=0
Oct 31 23:20:01.556982 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.556875    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"
Oct 31 23:20:01.556982 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.556918    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz" event={"ID":"3b0d47aa-36ab-46fa-b41f-20a71f2256cd","Type":"ContainerDied","Data":"8b7f17202fa4b84b3448c8fc2b9dff61a7e2c111c38a19277ce96ba1bea1125d"}
Oct 31 23:20:01.556982 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.556951    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz" event={"ID":"3b0d47aa-36ab-46fa-b41f-20a71f2256cd","Type":"ContainerDied","Data":"3f7a2fd73900e2cc79efa2b37dcf175a5a4f86068a67a5f4c2db63cc055207bb"}
Oct 31 23:20:01.569408 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.569388    4301 scope.go:122] "RemoveContainer" containerID="1a4188deb86605e67944e8210642d0dd1d1d233c923ccf29c897f6a81d5f1066"
Oct 31 23:20:01.569782 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:20:01.569763    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a4188deb86605e67944e8210642d0dd1d1d233c923ccf29c897f6a81d5f1066\": container with ID starting with 1a4188deb86605e67944e8210642d0dd1d1d233c923ccf29c897f6a81d5f1066 not found: ID does not exist" containerID="1a4188deb86605e67944e8210642d0dd1d1d233c923ccf29c897f6a81d5f1066"
Oct 31 23:20:01.569858 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.569789    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a4188deb86605e67944e8210642d0dd1d1d233c923ccf29c897f6a81d5f1066"} err="failed to get container status \"1a4188deb86605e67944e8210642d0dd1d1d233c923ccf29c897f6a81d5f1066\": rpc error: code = NotFound desc = could not find container \"1a4188deb86605e67944e8210642d0dd1d1d233c923ccf29c897f6a81d5f1066\": container with ID starting with 1a4188deb86605e67944e8210642d0dd1d1d233c923ccf29c897f6a81d5f1066 not found: ID does not exist"
Oct 31 23:20:01.569858 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.569807    4301 scope.go:122] "RemoveContainer" containerID="8b7f17202fa4b84b3448c8fc2b9dff61a7e2c111c38a19277ce96ba1bea1125d"
Oct 31 23:20:01.586249 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.586222    4301 scope.go:122] "RemoveContainer" containerID="8b7f17202fa4b84b3448c8fc2b9dff61a7e2c111c38a19277ce96ba1bea1125d"
Oct 31 23:20:01.587218 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:20:01.586831    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b7f17202fa4b84b3448c8fc2b9dff61a7e2c111c38a19277ce96ba1bea1125d\": container with ID starting with 8b7f17202fa4b84b3448c8fc2b9dff61a7e2c111c38a19277ce96ba1bea1125d not found: ID does not exist" containerID="8b7f17202fa4b84b3448c8fc2b9dff61a7e2c111c38a19277ce96ba1bea1125d"
Oct 31 23:20:01.587218 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.586890    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b7f17202fa4b84b3448c8fc2b9dff61a7e2c111c38a19277ce96ba1bea1125d"} err="failed to get container status \"8b7f17202fa4b84b3448c8fc2b9dff61a7e2c111c38a19277ce96ba1bea1125d\": rpc error: code = NotFound desc = could not find container \"8b7f17202fa4b84b3448c8fc2b9dff61a7e2c111c38a19277ce96ba1bea1125d\": container with ID starting with 8b7f17202fa4b84b3448c8fc2b9dff61a7e2c111c38a19277ce96ba1bea1125d not found: ID does not exist"
Oct 31 23:20:01.591616 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.591594    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"]
Oct 31 23:20:01.595487 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.595469    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-gvgnz"]
Oct 31 23:20:01.642762 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.642735    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-istio-data\") pod \"13c9f33c-af79-4938-b54f-142dc2f72218\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") "
Oct 31 23:20:01.642925 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.642788    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13c9f33c-af79-4938-b54f-142dc2f72218-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13c9f33c-af79-4938-b54f-142dc2f72218-istio-podinfo\") pod \"13c9f33c-af79-4938-b54f-142dc2f72218\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") "
Oct 31 23:20:01.642925 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.642827    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13c9f33c-af79-4938-b54f-142dc2f72218-istio-token\" (UniqueName: \"kubernetes.io/projected/13c9f33c-af79-4938-b54f-142dc2f72218-istio-token\") pod \"13c9f33c-af79-4938-b54f-142dc2f72218\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") "
Oct 31 23:20:01.643040 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.642952    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13c9f33c-af79-4938-b54f-142dc2f72218-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13c9f33c-af79-4938-b54f-142dc2f72218-istiod-ca-cert\") pod \"13c9f33c-af79-4938-b54f-142dc2f72218\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") "
Oct 31 23:20:01.643040 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.642993    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-workload-certs\") pod \"13c9f33c-af79-4938-b54f-142dc2f72218\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") "
Oct 31 23:20:01.643157 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.643032    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-istio-data" pod "13c9f33c-af79-4938-b54f-142dc2f72218" (UID: "13c9f33c-af79-4938-b54f-142dc2f72218"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:20:01.643157 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.643060    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-credential-socket\") pod \"13c9f33c-af79-4938-b54f-142dc2f72218\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") "
Oct 31 23:20:01.643268 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.643172    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13c9f33c-af79-4938-b54f-142dc2f72218-kube-api-access-nfd2t\" (UniqueName: \"kubernetes.io/projected/13c9f33c-af79-4938-b54f-142dc2f72218-kube-api-access-nfd2t\") pod \"13c9f33c-af79-4938-b54f-142dc2f72218\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") "
Oct 31 23:20:01.643268 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.643222    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-workload-socket\") pod \"13c9f33c-af79-4938-b54f-142dc2f72218\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") "
Oct 31 23:20:01.643268 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.643247    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-workload-certs" pod "13c9f33c-af79-4938-b54f-142dc2f72218" (UID: "13c9f33c-af79-4938-b54f-142dc2f72218"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:20:01.643480 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.643347    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-credential-socket" pod "13c9f33c-af79-4938-b54f-142dc2f72218" (UID: "13c9f33c-af79-4938-b54f-142dc2f72218"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:20:01.643480 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.643353    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-istio-envoy\") pod \"13c9f33c-af79-4938-b54f-142dc2f72218\" (UID: \"13c9f33c-af79-4938-b54f-142dc2f72218\") "
Oct 31 23:20:01.643480 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.643393    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13c9f33c-af79-4938-b54f-142dc2f72218-istiod-ca-cert" pod "13c9f33c-af79-4938-b54f-142dc2f72218" (UID: "13c9f33c-af79-4938-b54f-142dc2f72218"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:20:01.643655 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.643640    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-workload-socket" pod "13c9f33c-af79-4938-b54f-142dc2f72218" (UID: "13c9f33c-af79-4938-b54f-142dc2f72218"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:20:01.644438 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.644412    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:20:01.644550 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.644440    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13c9f33c-af79-4938-b54f-142dc2f72218-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:20:01.644550 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.644457    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:20:01.644550 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.644470    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:20:01.644550 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.644485    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:20:01.645381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.645346    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13c9f33c-af79-4938-b54f-142dc2f72218-kube-api-access-nfd2t" pod "13c9f33c-af79-4938-b54f-142dc2f72218" (UID: "13c9f33c-af79-4938-b54f-142dc2f72218"). InnerVolumeSpecName "kube-api-access-nfd2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:20:01.645461 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.645394    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13c9f33c-af79-4938-b54f-142dc2f72218-istio-token" pod "13c9f33c-af79-4938-b54f-142dc2f72218" (UID: "13c9f33c-af79-4938-b54f-142dc2f72218"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:20:01.645588 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.645568    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13c9f33c-af79-4938-b54f-142dc2f72218-istio-podinfo" pod "13c9f33c-af79-4938-b54f-142dc2f72218" (UID: "13c9f33c-af79-4938-b54f-142dc2f72218"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:20:01.645651 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.645610    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-istio-envoy" pod "13c9f33c-af79-4938-b54f-142dc2f72218" (UID: "13c9f33c-af79-4938-b54f-142dc2f72218"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:20:01.745854 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.745773    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfd2t\" (UniqueName: \"kubernetes.io/projected/13c9f33c-af79-4938-b54f-142dc2f72218-kube-api-access-nfd2t\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:20:01.745854 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.745803    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13c9f33c-af79-4938-b54f-142dc2f72218-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:20:01.745854 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.745815    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13c9f33c-af79-4938-b54f-142dc2f72218-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:20:01.745854 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.745824    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13c9f33c-af79-4938-b54f-142dc2f72218-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:20:01.884150 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.884111    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"]
Oct 31 23:20:01.888189 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:01.888162    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8ztkz"]
Oct 31 23:20:02.563603 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:20:02.563565    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:20:02.959417 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:02.959373    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:20:02.959606 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:02.959476    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:20:02.959606 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:02.959528    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:20:03.407550 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:03.407517    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13c9f33c-af79-4938-b54f-142dc2f72218" path="/var/lib/kubelet/pods/13c9f33c-af79-4938-b54f-142dc2f72218/volumes"
Oct 31 23:20:03.408095 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:03.408079    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b0d47aa-36ab-46fa-b41f-20a71f2256cd" path="/var/lib/kubelet/pods/3b0d47aa-36ab-46fa-b41f-20a71f2256cd/volumes"
Oct 31 23:20:03.623458 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:03.623421    4301 generic.go:356] "Generic (PLEG): container finished" podID="f7560a60-cbba-43ba-81a1-946c57663b80" containerID="b711c656e8708f7a548a7453b46176951c1341be574dfd1c54c987141351ef13" exitCode=0
Oct 31 23:20:03.623861 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:03.623506    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" event={"ID":"f7560a60-cbba-43ba-81a1-946c57663b80","Type":"ContainerDied","Data":"b711c656e8708f7a548a7453b46176951c1341be574dfd1c54c987141351ef13"}
Oct 31 23:20:03.623861 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:03.623572    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" event={"ID":"f7560a60-cbba-43ba-81a1-946c57663b80","Type":"ContainerStarted","Data":"aa732f12d6a9ce2ff5978da02c04bf10f8256e93d9e297a2cbd2d997864ada20"}
Oct 31 23:20:03.838919 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:03.838837    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:20:04.897475 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:04.894735    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:20:05.647054 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:05.647014    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:20:05.648014 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:05.647992    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:20:17.357083 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:17.357052    4301 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vc2j7_5f102a31-27f2-410b-9e36-2cdb8d9e852c/konnectivity-agent/0.log" limit=1024
Oct 31 23:20:18.369480 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:18.369439    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85b90bac-ghwkw"]
Oct 31 23:20:18.371118 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:18.371084    4301 cpu_manager.go:414] "containerMap: removing container" podUID="13c9f33c-af79-4938-b54f-142dc2f72218" containerName="istio-proxy"
Oct 31 23:20:18.371410 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:18.371389    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13c9f33c-af79-4938-b54f-142dc2f72218" containerName="istio-proxy"
Oct 31 23:20:18.371576 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:18.371563    4301 cpu_manager.go:414] "containerMap: removing container" podUID="3b0d47aa-36ab-46fa-b41f-20a71f2256cd" containerName="istio-proxy"
Oct 31 23:20:18.371656 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:18.371645    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b0d47aa-36ab-46fa-b41f-20a71f2256cd" containerName="istio-proxy"
Oct 31 23:20:18.371734 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:18.371724    4301 cpu_manager.go:414] "containerMap: removing container" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main"
Oct 31 23:20:18.371804 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:18.371794    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main"
Oct 31 23:20:18.372113 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:18.372098    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="13c9f33c-af79-4938-b54f-142dc2f72218" containerName="istio-proxy"
Oct 31 23:20:18.385727 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:18.385692    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b0d47aa-36ab-46fa-b41f-20a71f2256cd" containerName="istio-proxy"
Oct 31 23:20:18.385964 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:18.385949    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff4f6df9-e1f0-4163-895c-7df6b065b033" containerName="main"
Oct 31 23:20:18.403083 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:18.403055    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85b90bac-ghwkw"]
Oct 31 23:20:18.413012 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:18.404106    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85b90bac-ghwkw"
Oct 31 23:20:18.425379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:18.420645    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 23:20:18.425379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:18.420782    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\""
Oct 31 23:20:18.505815 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:18.505766    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfd9aa59-6685-4b51-976a-cd4dfe5802b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-85b90bac-ghwkw\" (UID: \"cfd9aa59-6685-4b51-976a-cd4dfe5802b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85b90bac-ghwkw"
Oct 31 23:20:18.506014 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:18.505885    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r56q\" (UniqueName: \"kubernetes.io/projected/cfd9aa59-6685-4b51-976a-cd4dfe5802b4-kube-api-access-9r56q\") pod \"maas-api-revoke-keys-e2e-mt-85b90bac-ghwkw\" (UID: \"cfd9aa59-6685-4b51-976a-cd4dfe5802b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85b90bac-ghwkw"
Oct 31 23:20:18.506075 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:18.506042    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfd9aa59-6685-4b51-976a-cd4dfe5802b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-85b90bac-ghwkw\" (UID: \"cfd9aa59-6685-4b51-976a-cd4dfe5802b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85b90bac-ghwkw"
Oct 31 23:20:18.607312 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:18.607265    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r56q\" (UniqueName: \"kubernetes.io/projected/cfd9aa59-6685-4b51-976a-cd4dfe5802b4-kube-api-access-9r56q\") pod \"maas-api-revoke-keys-e2e-mt-85b90bac-ghwkw\" (UID: \"cfd9aa59-6685-4b51-976a-cd4dfe5802b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85b90bac-ghwkw"
Oct 31 23:20:18.607536 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:18.607453    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfd9aa59-6685-4b51-976a-cd4dfe5802b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-85b90bac-ghwkw\" (UID: \"cfd9aa59-6685-4b51-976a-cd4dfe5802b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85b90bac-ghwkw"
Oct 31 23:20:18.607688 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:18.607666    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfd9aa59-6685-4b51-976a-cd4dfe5802b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-85b90bac-ghwkw\" (UID: \"cfd9aa59-6685-4b51-976a-cd4dfe5802b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85b90bac-ghwkw"
Oct 31 23:20:18.608454 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:18.608428    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfd9aa59-6685-4b51-976a-cd4dfe5802b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-85b90bac-ghwkw\" (UID: \"cfd9aa59-6685-4b51-976a-cd4dfe5802b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85b90bac-ghwkw"
Oct 31 23:20:18.610313 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:18.610288    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfd9aa59-6685-4b51-976a-cd4dfe5802b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-85b90bac-ghwkw\" (UID: \"cfd9aa59-6685-4b51-976a-cd4dfe5802b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85b90bac-ghwkw"
Oct 31 23:20:18.619021 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:18.618997    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r56q\" (UniqueName: \"kubernetes.io/projected/cfd9aa59-6685-4b51-976a-cd4dfe5802b4-kube-api-access-9r56q\") pod \"maas-api-revoke-keys-e2e-mt-85b90bac-ghwkw\" (UID: \"cfd9aa59-6685-4b51-976a-cd4dfe5802b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85b90bac-ghwkw"
Oct 31 23:20:18.752897 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:18.752813    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85b90bac-ghwkw"
Oct 31 23:20:19.072984 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:19.072870    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85b90bac-ghwkw"]
Oct 31 23:20:19.087378 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:20:19.084035    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd9aa59_6685_4b51_976a_cd4dfe5802b4.slice/crio-578a9f09b5c73431528c9fef8df4097031c218d01d134e7fb5f0bedcabd4fccb WatchSource:0}: Error finding container 578a9f09b5c73431528c9fef8df4097031c218d01d134e7fb5f0bedcabd4fccb: Status 404 returned error can't find the container with id 578a9f09b5c73431528c9fef8df4097031c218d01d134e7fb5f0bedcabd4fccb
Oct 31 23:20:19.887341 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:19.887297    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85b90bac-ghwkw" event={"ID":"cfd9aa59-6685-4b51-976a-cd4dfe5802b4","Type":"ContainerStarted","Data":"d4cca9d627153fe04dd17cc11e7b3165096b3c36e35ea945cf0ec7a47fb5055f"}
Oct 31 23:20:19.887341 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:19.887367    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85b90bac-ghwkw" event={"ID":"cfd9aa59-6685-4b51-976a-cd4dfe5802b4","Type":"ContainerStarted","Data":"578a9f09b5c73431528c9fef8df4097031c218d01d134e7fb5f0bedcabd4fccb"}
Oct 31 23:20:19.907914 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:19.907851    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85b90bac-ghwkw" podStartSLOduration=1.907832907 podStartE2EDuration="1.907832907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:20:19.903469825 +0000 UTC m=+1838.650659329" watchObservedRunningTime="2026-10-31 23:20:19.907832907 +0000 UTC m=+1838.655022414"
Oct 31 23:20:20.898377 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:20.898317    4301 generic.go:356] "Generic (PLEG): container finished" podID="cfd9aa59-6685-4b51-976a-cd4dfe5802b4" containerID="d4cca9d627153fe04dd17cc11e7b3165096b3c36e35ea945cf0ec7a47fb5055f" exitCode=0
Oct 31 23:20:20.898965 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:20.898441    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85b90bac-ghwkw" event={"ID":"cfd9aa59-6685-4b51-976a-cd4dfe5802b4","Type":"ContainerDied","Data":"d4cca9d627153fe04dd17cc11e7b3165096b3c36e35ea945cf0ec7a47fb5055f"}
Oct 31 23:20:22.151422 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:22.151380    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85b90bac-ghwkw"
Oct 31 23:20:22.237804 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:22.237714    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfd9aa59-6685-4b51-976a-cd4dfe5802b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfd9aa59-6685-4b51-976a-cd4dfe5802b4-maas-api-openshift-ca\") pod \"cfd9aa59-6685-4b51-976a-cd4dfe5802b4\" (UID: \"cfd9aa59-6685-4b51-976a-cd4dfe5802b4\") "
Oct 31 23:20:22.239012 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:22.237872    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfd9aa59-6685-4b51-976a-cd4dfe5802b4-kube-api-access-9r56q\" (UniqueName: \"kubernetes.io/projected/cfd9aa59-6685-4b51-976a-cd4dfe5802b4-kube-api-access-9r56q\") pod \"cfd9aa59-6685-4b51-976a-cd4dfe5802b4\" (UID: \"cfd9aa59-6685-4b51-976a-cd4dfe5802b4\") "
Oct 31 23:20:22.239012 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:22.238001    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfd9aa59-6685-4b51-976a-cd4dfe5802b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfd9aa59-6685-4b51-976a-cd4dfe5802b4-maas-api-xks-ca\") pod \"cfd9aa59-6685-4b51-976a-cd4dfe5802b4\" (UID: \"cfd9aa59-6685-4b51-976a-cd4dfe5802b4\") "
Oct 31 23:20:22.239012 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:22.238400    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfd9aa59-6685-4b51-976a-cd4dfe5802b4-maas-api-openshift-ca" pod "cfd9aa59-6685-4b51-976a-cd4dfe5802b4" (UID: "cfd9aa59-6685-4b51-976a-cd4dfe5802b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:20:22.239012 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:22.238929    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfd9aa59-6685-4b51-976a-cd4dfe5802b4-maas-api-openshift-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:20:22.241833 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:22.241779    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd9aa59-6685-4b51-976a-cd4dfe5802b4-kube-api-access-9r56q" pod "cfd9aa59-6685-4b51-976a-cd4dfe5802b4" (UID: "cfd9aa59-6685-4b51-976a-cd4dfe5802b4"). InnerVolumeSpecName "kube-api-access-9r56q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:20:22.244426 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:22.244387    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfd9aa59-6685-4b51-976a-cd4dfe5802b4-maas-api-xks-ca" pod "cfd9aa59-6685-4b51-976a-cd4dfe5802b4" (UID: "cfd9aa59-6685-4b51-976a-cd4dfe5802b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:20:22.340784 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:22.340751    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfd9aa59-6685-4b51-976a-cd4dfe5802b4-maas-api-xks-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:20:22.340784 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:22.340779    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r56q\" (UniqueName: \"kubernetes.io/projected/cfd9aa59-6685-4b51-976a-cd4dfe5802b4-kube-api-access-9r56q\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:20:22.915949 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:22.915916    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85b90bac-ghwkw" event={"ID":"cfd9aa59-6685-4b51-976a-cd4dfe5802b4","Type":"ContainerDied","Data":"578a9f09b5c73431528c9fef8df4097031c218d01d134e7fb5f0bedcabd4fccb"}
Oct 31 23:20:22.916121 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:22.915963    4301 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="578a9f09b5c73431528c9fef8df4097031c218d01d134e7fb5f0bedcabd4fccb"
Oct 31 23:20:22.916121 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:22.915928    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85b90bac-ghwkw"
Oct 31 23:20:25.557327 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:25.557244    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-479107-748df6cc67-5hrhg"]
Oct 31 23:20:25.558117 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:25.558099    4301 cpu_manager.go:414] "containerMap: removing container" podUID="cfd9aa59-6685-4b51-976a-cd4dfe5802b4" containerName="revoke-keys"
Oct 31 23:20:25.558169 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:25.558119    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfd9aa59-6685-4b51-976a-cd4dfe5802b4" containerName="revoke-keys"
Oct 31 23:20:25.558219 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:25.558210    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfd9aa59-6685-4b51-976a-cd4dfe5802b4" containerName="revoke-keys"
Oct 31 23:20:25.563903 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:25.563875    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-479107-748df6cc67-5hrhg"
Oct 31 23:20:25.587636 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:25.587607    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-479107\""
Oct 31 23:20:25.590828 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:25.590801    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-479107-748df6cc67-5hrhg"]
Oct 31 23:20:25.605026 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:25.604984    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb1cddef-0805-49a6-86dc-4489565710bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-479107-748df6cc67-5hrhg\" (UID: \"bb1cddef-0805-49a6-86dc-4489565710bd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-479107-748df6cc67-5hrhg"
Oct 31 23:20:25.605293 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:25.605273    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfdfw\" (UniqueName: \"kubernetes.io/projected/bb1cddef-0805-49a6-86dc-4489565710bd-kube-api-access-lfdfw\") pod \"payload-pre-processing-e2e-api-a-479107-748df6cc67-5hrhg\" (UID: \"bb1cddef-0805-49a6-86dc-4489565710bd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-479107-748df6cc67-5hrhg"
Oct 31 23:20:25.629982 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:25.629941    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-479107-6d88b6b77d-hzkkg"]
Oct 31 23:20:25.634757 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:25.634727    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-479107-6d88b6b77d-hzkkg"
Oct 31 23:20:25.643437 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:25.643407    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-479107-dockercfg-p2lm8\""
Oct 31 23:20:25.707093 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:25.707038    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfdfw\" (UniqueName: \"kubernetes.io/projected/bb1cddef-0805-49a6-86dc-4489565710bd-kube-api-access-lfdfw\") pod \"payload-pre-processing-e2e-api-a-479107-748df6cc67-5hrhg\" (UID: \"bb1cddef-0805-49a6-86dc-4489565710bd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-479107-748df6cc67-5hrhg"
Oct 31 23:20:25.707329 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:25.707115    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfqgj\" (UniqueName: \"kubernetes.io/projected/cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27-kube-api-access-hfqgj\") pod \"payload-processing-e2e-api-a-479107-6d88b6b77d-hzkkg\" (UID: \"cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27\") " pod="openshift-ingress/payload-processing-e2e-api-a-479107-6d88b6b77d-hzkkg"
Oct 31 23:20:25.707468 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:25.707351    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27-plugins-config-volume\") pod \"payload-processing-e2e-api-a-479107-6d88b6b77d-hzkkg\" (UID: \"cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27\") " pod="openshift-ingress/payload-processing-e2e-api-a-479107-6d88b6b77d-hzkkg"
Oct 31 23:20:25.707468 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:25.707430    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb1cddef-0805-49a6-86dc-4489565710bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-479107-748df6cc67-5hrhg\" (UID: \"bb1cddef-0805-49a6-86dc-4489565710bd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-479107-748df6cc67-5hrhg"
Oct 31 23:20:25.708456 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:25.708429    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb1cddef-0805-49a6-86dc-4489565710bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-479107-748df6cc67-5hrhg\" (UID: \"bb1cddef-0805-49a6-86dc-4489565710bd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-479107-748df6cc67-5hrhg"
Oct 31 23:20:25.708975 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:25.708943    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-479107-6d88b6b77d-hzkkg"]
Oct 31 23:20:25.732557 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:25.732497    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfdfw\" (UniqueName: \"kubernetes.io/projected/bb1cddef-0805-49a6-86dc-4489565710bd-kube-api-access-lfdfw\") pod \"payload-pre-processing-e2e-api-a-479107-748df6cc67-5hrhg\" (UID: \"bb1cddef-0805-49a6-86dc-4489565710bd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-479107-748df6cc67-5hrhg"
Oct 31 23:20:26.011415 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:25.908084    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfqgj\" (UniqueName: \"kubernetes.io/projected/cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27-kube-api-access-hfqgj\") pod \"payload-processing-e2e-api-a-479107-6d88b6b77d-hzkkg\" (UID: \"cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27\") " pod="openshift-ingress/payload-processing-e2e-api-a-479107-6d88b6b77d-hzkkg"
Oct 31 23:20:26.054538 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:26.014946    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-479107-748df6cc67-5hrhg"
Oct 31 23:20:26.189233 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:26.054964    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27-plugins-config-volume\") pod \"payload-processing-e2e-api-a-479107-6d88b6b77d-hzkkg\" (UID: \"cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27\") " pod="openshift-ingress/payload-processing-e2e-api-a-479107-6d88b6b77d-hzkkg"
Oct 31 23:20:26.189233 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:26.100423    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27-plugins-config-volume\") pod \"payload-processing-e2e-api-a-479107-6d88b6b77d-hzkkg\" (UID: \"cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27\") " pod="openshift-ingress/payload-processing-e2e-api-a-479107-6d88b6b77d-hzkkg"
Oct 31 23:20:26.189233 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:26.166746    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfqgj\" (UniqueName: \"kubernetes.io/projected/cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27-kube-api-access-hfqgj\") pod \"payload-processing-e2e-api-a-479107-6d88b6b77d-hzkkg\" (UID: \"cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27\") " pod="openshift-ingress/payload-processing-e2e-api-a-479107-6d88b6b77d-hzkkg"
Oct 31 23:20:26.261394 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:26.258029    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-479107-6d88b6b77d-hzkkg"
Oct 31 23:20:26.510804 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:26.510771    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-479107-748df6cc67-5hrhg"]
Oct 31 23:20:26.516168 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:20:26.516123    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb1cddef_0805_49a6_86dc_4489565710bd.slice/crio-5b52177371ee4f02c03ca38bed43de4c247420ecf71e9decaf12ea2c0955e371 WatchSource:0}: Error finding container 5b52177371ee4f02c03ca38bed43de4c247420ecf71e9decaf12ea2c0955e371: Status 404 returned error can't find the container with id 5b52177371ee4f02c03ca38bed43de4c247420ecf71e9decaf12ea2c0955e371
Oct 31 23:20:26.579396 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:26.579366    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-479107-6d88b6b77d-hzkkg"]
Oct 31 23:20:26.583907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:26.583871    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4528ed-5446b78459-n8x8d"]
Oct 31 23:20:26.585684 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:20:26.585664    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcabc6ea0_3cd9_4a11_bb51_d674b3dd7a27.slice/crio-ea1bb8a6d794a6931d0b17d6f279facc19bff7c9d87ed5b052d570a7458e29e5 WatchSource:0}: Error finding container ea1bb8a6d794a6931d0b17d6f279facc19bff7c9d87ed5b052d570a7458e29e5: Status 404 returned error can't find the container with id ea1bb8a6d794a6931d0b17d6f279facc19bff7c9d87ed5b052d570a7458e29e5
Oct 31 23:20:26.589078 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:26.589059    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4528ed-5446b78459-n8x8d"
Oct 31 23:20:26.593618 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:26.593577    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk4tp\" (UniqueName: \"kubernetes.io/projected/9ec0f289-97b7-45f0-952c-309c3b97d414-kube-api-access-fk4tp\") pod \"payload-pre-processing-e2e-api-a-4528ed-5446b78459-n8x8d\" (UID: \"9ec0f289-97b7-45f0-952c-309c3b97d414\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4528ed-5446b78459-n8x8d"
Oct 31 23:20:26.593929 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:26.593912    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ec0f289-97b7-45f0-952c-309c3b97d414-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4528ed-5446b78459-n8x8d\" (UID: \"9ec0f289-97b7-45f0-952c-309c3b97d414\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4528ed-5446b78459-n8x8d"
Oct 31 23:20:26.598481 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:26.597517    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-4528ed\""
Oct 31 23:20:26.598481 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:26.597864    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-4528ed-dockercfg-nf5rs\""
Oct 31 23:20:26.608617 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:26.608583    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4528ed-5446b78459-n8x8d"]
Oct 31 23:20:26.608769 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:26.608661    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4528ed-64b678979d-spcpj"]
Oct 31 23:20:26.620450 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:26.620426    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4528ed-64b678979d-spcpj"
Oct 31 23:20:26.625039 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:26.625016    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4528ed-64b678979d-spcpj"]
Oct 31 23:20:26.718409 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:26.718079    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk4tp\" (UniqueName: \"kubernetes.io/projected/9ec0f289-97b7-45f0-952c-309c3b97d414-kube-api-access-fk4tp\") pod \"payload-pre-processing-e2e-api-a-4528ed-5446b78459-n8x8d\" (UID: \"9ec0f289-97b7-45f0-952c-309c3b97d414\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4528ed-5446b78459-n8x8d"
Oct 31 23:20:26.718409 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:26.718257    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3756c42-084c-4986-84f3-720fcf5ad4e1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4528ed-64b678979d-spcpj\" (UID: \"a3756c42-084c-4986-84f3-720fcf5ad4e1\") " pod="openshift-ingress/payload-processing-e2e-api-a-4528ed-64b678979d-spcpj"
Oct 31 23:20:26.718666 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:26.718477    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6cqv\" (UniqueName: \"kubernetes.io/projected/a3756c42-084c-4986-84f3-720fcf5ad4e1-kube-api-access-b6cqv\") pod \"payload-processing-e2e-api-a-4528ed-64b678979d-spcpj\" (UID: \"a3756c42-084c-4986-84f3-720fcf5ad4e1\") " pod="openshift-ingress/payload-processing-e2e-api-a-4528ed-64b678979d-spcpj"
Oct 31 23:20:26.726391 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:26.718727    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ec0f289-97b7-45f0-952c-309c3b97d414-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4528ed-5446b78459-n8x8d\" (UID: \"9ec0f289-97b7-45f0-952c-309c3b97d414\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4528ed-5446b78459-n8x8d"
Oct 31 23:20:26.752417 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:26.737735    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ec0f289-97b7-45f0-952c-309c3b97d414-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4528ed-5446b78459-n8x8d\" (UID: \"9ec0f289-97b7-45f0-952c-309c3b97d414\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4528ed-5446b78459-n8x8d"
Oct 31 23:20:26.779444 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:26.767011    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk4tp\" (UniqueName: \"kubernetes.io/projected/9ec0f289-97b7-45f0-952c-309c3b97d414-kube-api-access-fk4tp\") pod \"payload-pre-processing-e2e-api-a-4528ed-5446b78459-n8x8d\" (UID: \"9ec0f289-97b7-45f0-952c-309c3b97d414\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4528ed-5446b78459-n8x8d"
Oct 31 23:20:26.835175 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:26.835138    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3756c42-084c-4986-84f3-720fcf5ad4e1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4528ed-64b678979d-spcpj\" (UID: \"a3756c42-084c-4986-84f3-720fcf5ad4e1\") " pod="openshift-ingress/payload-processing-e2e-api-a-4528ed-64b678979d-spcpj"
Oct 31 23:20:26.836657 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:26.836473    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3756c42-084c-4986-84f3-720fcf5ad4e1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4528ed-64b678979d-spcpj\" (UID: \"a3756c42-084c-4986-84f3-720fcf5ad4e1\") " pod="openshift-ingress/payload-processing-e2e-api-a-4528ed-64b678979d-spcpj"
Oct 31 23:20:26.837206 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:26.837181    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6cqv\" (UniqueName: \"kubernetes.io/projected/a3756c42-084c-4986-84f3-720fcf5ad4e1-kube-api-access-b6cqv\") pod \"payload-processing-e2e-api-a-4528ed-64b678979d-spcpj\" (UID: \"a3756c42-084c-4986-84f3-720fcf5ad4e1\") " pod="openshift-ingress/payload-processing-e2e-api-a-4528ed-64b678979d-spcpj"
Oct 31 23:20:26.865276 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:26.865204    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6cqv\" (UniqueName: \"kubernetes.io/projected/a3756c42-084c-4986-84f3-720fcf5ad4e1-kube-api-access-b6cqv\") pod \"payload-processing-e2e-api-a-4528ed-64b678979d-spcpj\" (UID: \"a3756c42-084c-4986-84f3-720fcf5ad4e1\") " pod="openshift-ingress/payload-processing-e2e-api-a-4528ed-64b678979d-spcpj"
Oct 31 23:20:26.928448 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:26.914700    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4528ed-5446b78459-n8x8d"
Oct 31 23:20:26.949940 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:26.949695    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4528ed-64b678979d-spcpj"
Oct 31 23:20:27.161432 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:27.158641    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-479107-748df6cc67-5hrhg" event={"ID":"bb1cddef-0805-49a6-86dc-4489565710bd","Type":"ContainerStarted","Data":"8c87951594dd210ed9fb07437565df5387aca7630f3615a4afdde98d9ee2e26d"}
Oct 31 23:20:27.161432 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:27.158711    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-479107-748df6cc67-5hrhg" event={"ID":"bb1cddef-0805-49a6-86dc-4489565710bd","Type":"ContainerStarted","Data":"5b52177371ee4f02c03ca38bed43de4c247420ecf71e9decaf12ea2c0955e371"}
Oct 31 23:20:27.161432 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:27.159568    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-479107-748df6cc67-5hrhg"
Oct 31 23:20:27.163874 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:27.162120    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-479107-6d88b6b77d-hzkkg" event={"ID":"cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27","Type":"ContainerStarted","Data":"cc0d97e3dcacb25b42b3d6d5870bef3b196bdbe80c6f38266d7919e08bfc2b13"}
Oct 31 23:20:27.163874 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:27.162174    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-479107-6d88b6b77d-hzkkg" event={"ID":"cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27","Type":"ContainerStarted","Data":"ea1bb8a6d794a6931d0b17d6f279facc19bff7c9d87ed5b052d570a7458e29e5"}
Oct 31 23:20:27.163874 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:27.162896    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-479107-6d88b6b77d-hzkkg"
Oct 31 23:20:27.192446 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:27.191982    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-479107-748df6cc67-5hrhg" podStartSLOduration=2.191958141 podStartE2EDuration="2.191958141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:20:27.184929779 +0000 UTC m=+1845.932119286" watchObservedRunningTime="2026-10-31 23:20:27.191958141 +0000 UTC m=+1845.939147646"
Oct 31 23:20:27.233381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:27.233282    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-479107-6d88b6b77d-hzkkg" podStartSLOduration=2.233265537 podStartE2EDuration="2.233265537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:20:27.232664465 +0000 UTC m=+1845.979853968" watchObservedRunningTime="2026-10-31 23:20:27.233265537 +0000 UTC m=+1845.980455045"
Oct 31 23:20:27.483379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:27.476704    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4528ed-64b678979d-spcpj"]
Oct 31 23:20:27.502704 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:20:27.502618    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3756c42_084c_4986_84f3_720fcf5ad4e1.slice/crio-bb5590d41dccdb9b4617c1502f6eba05b947a1471f8325e456d11a026d79fd30 WatchSource:0}: Error finding container bb5590d41dccdb9b4617c1502f6eba05b947a1471f8325e456d11a026d79fd30: Status 404 returned error can't find the container with id bb5590d41dccdb9b4617c1502f6eba05b947a1471f8325e456d11a026d79fd30
Oct 31 23:20:27.554990 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:20:27.554851    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec0f289_97b7_45f0_952c_309c3b97d414.slice/crio-ea2cb1e6efc0d8af219b1b449823a6a8ef1da2e2c034181d75d2d658933a13e1 WatchSource:0}: Error finding container ea2cb1e6efc0d8af219b1b449823a6a8ef1da2e2c034181d75d2d658933a13e1: Status 404 returned error can't find the container with id ea2cb1e6efc0d8af219b1b449823a6a8ef1da2e2c034181d75d2d658933a13e1
Oct 31 23:20:27.556533 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:27.556510    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4528ed-5446b78459-n8x8d"]
Oct 31 23:20:28.172644 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:28.172587    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4528ed-64b678979d-spcpj" event={"ID":"a3756c42-084c-4986-84f3-720fcf5ad4e1","Type":"ContainerStarted","Data":"4d60c7c010310bb505d3b83f26b669716fbace36a44c9c1b2eabe37f7eec344a"}
Oct 31 23:20:28.172644 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:28.172642    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4528ed-64b678979d-spcpj" event={"ID":"a3756c42-084c-4986-84f3-720fcf5ad4e1","Type":"ContainerStarted","Data":"bb5590d41dccdb9b4617c1502f6eba05b947a1471f8325e456d11a026d79fd30"}
Oct 31 23:20:28.173072 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:28.172684    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-4528ed-64b678979d-spcpj"
Oct 31 23:20:28.174332 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:28.174307    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4528ed-5446b78459-n8x8d" event={"ID":"9ec0f289-97b7-45f0-952c-309c3b97d414","Type":"ContainerStarted","Data":"f6fb7fed841e3be3031a82bad57f808919bec5be3a0e37bdf65a4799c580cece"}
Oct 31 23:20:28.174478 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:28.174352    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4528ed-5446b78459-n8x8d" event={"ID":"9ec0f289-97b7-45f0-952c-309c3b97d414","Type":"ContainerStarted","Data":"ea2cb1e6efc0d8af219b1b449823a6a8ef1da2e2c034181d75d2d658933a13e1"}
Oct 31 23:20:28.174478 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:28.174417    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4528ed-5446b78459-n8x8d"
Oct 31 23:20:28.237331 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:28.237275    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-4528ed-64b678979d-spcpj" podStartSLOduration=2.237261666 podStartE2EDuration="2.237261666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:20:28.23575496 +0000 UTC m=+1846.982944463" watchObservedRunningTime="2026-10-31 23:20:28.237261666 +0000 UTC m=+1846.984451213"
Oct 31 23:20:28.269482 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:28.269436    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4528ed-5446b78459-n8x8d" podStartSLOduration=2.269422957 podStartE2EDuration="2.269422957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:20:28.265743184 +0000 UTC m=+1847.012932689" watchObservedRunningTime="2026-10-31 23:20:28.269422957 +0000 UTC m=+1847.016612462"
Oct 31 23:20:28.586559 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:28.586479    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-85b90bac-bcdc55f5f-vtqx4"]
Oct 31 23:20:32.550714 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:20:32.550670    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:20:33.587265 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:33.587224    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-85b90bac-bcdc55f5f-vtqx4" podUID="ad66adfb-6f3e-43bf-8499-4b63285c1849" containerName="payload-pre-processing" containerID="cri-o://6f437b34b83e514b19c4a38abe51f1ceb7adbf74858311dc64c44f5a4646e08e" gracePeriod=25
Oct 31 23:20:33.878476 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:33.878449    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-85b90bac-bcdc55f5f-vtqx4"
Oct 31 23:20:33.882160 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:33.882134    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad66adfb-6f3e-43bf-8499-4b63285c1849-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad66adfb-6f3e-43bf-8499-4b63285c1849-plugins-config-volume\") pod \"ad66adfb-6f3e-43bf-8499-4b63285c1849\" (UID: \"ad66adfb-6f3e-43bf-8499-4b63285c1849\") "
Oct 31 23:20:33.882291 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:33.882228    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad66adfb-6f3e-43bf-8499-4b63285c1849-kube-api-access-vxpnk\" (UniqueName: \"kubernetes.io/projected/ad66adfb-6f3e-43bf-8499-4b63285c1849-kube-api-access-vxpnk\") pod \"ad66adfb-6f3e-43bf-8499-4b63285c1849\" (UID: \"ad66adfb-6f3e-43bf-8499-4b63285c1849\") "
Oct 31 23:20:33.882641 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:33.882618    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad66adfb-6f3e-43bf-8499-4b63285c1849-plugins-config-volume" pod "ad66adfb-6f3e-43bf-8499-4b63285c1849" (UID: "ad66adfb-6f3e-43bf-8499-4b63285c1849"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:20:33.882900 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:33.882884    4301 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad66adfb-6f3e-43bf-8499-4b63285c1849-plugins-config-volume\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:20:33.884342 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:33.884325    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad66adfb-6f3e-43bf-8499-4b63285c1849-kube-api-access-vxpnk" pod "ad66adfb-6f3e-43bf-8499-4b63285c1849" (UID: "ad66adfb-6f3e-43bf-8499-4b63285c1849"). InnerVolumeSpecName "kube-api-access-vxpnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:20:34.037378 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:34.008306    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxpnk\" (UniqueName: \"kubernetes.io/projected/ad66adfb-6f3e-43bf-8499-4b63285c1849-kube-api-access-vxpnk\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:20:34.347864 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:34.347827    4301 generic.go:356] "Generic (PLEG): container finished" podID="ad66adfb-6f3e-43bf-8499-4b63285c1849" containerID="6f437b34b83e514b19c4a38abe51f1ceb7adbf74858311dc64c44f5a4646e08e" exitCode=0
Oct 31 23:20:34.348038 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:34.347879    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-85b90bac-bcdc55f5f-vtqx4"
Oct 31 23:20:34.348038 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:34.347913    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-85b90bac-bcdc55f5f-vtqx4" event={"ID":"ad66adfb-6f3e-43bf-8499-4b63285c1849","Type":"ContainerDied","Data":"6f437b34b83e514b19c4a38abe51f1ceb7adbf74858311dc64c44f5a4646e08e"}
Oct 31 23:20:34.348038 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:34.347972    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-85b90bac-bcdc55f5f-vtqx4" event={"ID":"ad66adfb-6f3e-43bf-8499-4b63285c1849","Type":"ContainerDied","Data":"0e4c6b0b7d5de81381f8d033a93375ca9af81bfbd437d429b99799914624fd7e"}
Oct 31 23:20:34.348038 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:34.347995    4301 scope.go:122] "RemoveContainer" containerID="6f437b34b83e514b19c4a38abe51f1ceb7adbf74858311dc64c44f5a4646e08e"
Oct 31 23:20:34.363421 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:34.363342    4301 scope.go:122] "RemoveContainer" containerID="6f437b34b83e514b19c4a38abe51f1ceb7adbf74858311dc64c44f5a4646e08e"
Oct 31 23:20:34.363694 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:20:34.363674    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f437b34b83e514b19c4a38abe51f1ceb7adbf74858311dc64c44f5a4646e08e\": container with ID starting with 6f437b34b83e514b19c4a38abe51f1ceb7adbf74858311dc64c44f5a4646e08e not found: ID does not exist" containerID="6f437b34b83e514b19c4a38abe51f1ceb7adbf74858311dc64c44f5a4646e08e"
Oct 31 23:20:34.363766 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:34.363704    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f437b34b83e514b19c4a38abe51f1ceb7adbf74858311dc64c44f5a4646e08e"} err="failed to get container status \"6f437b34b83e514b19c4a38abe51f1ceb7adbf74858311dc64c44f5a4646e08e\": rpc error: code = NotFound desc = could not find container \"6f437b34b83e514b19c4a38abe51f1ceb7adbf74858311dc64c44f5a4646e08e\": container with ID starting with 6f437b34b83e514b19c4a38abe51f1ceb7adbf74858311dc64c44f5a4646e08e not found: ID does not exist"
Oct 31 23:20:34.372770 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:34.372745    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-85b90bac-bcdc55f5f-vtqx4"]
Oct 31 23:20:34.377062 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:34.377039    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-85b90bac-bcdc55f5f-vtqx4"]
Oct 31 23:20:35.464227 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:35.464183    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad66adfb-6f3e-43bf-8499-4b63285c1849" path="/var/lib/kubelet/pods/ad66adfb-6f3e-43bf-8499-4b63285c1849/volumes"
Oct 31 23:20:38.178827 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:38.178797    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-479107-748df6cc67-5hrhg"
Oct 31 23:20:39.297423 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:39.293575    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-4528ed-64b678979d-spcpj"
Oct 31 23:20:39.297423 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:39.297404    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4528ed-5446b78459-n8x8d"
Oct 31 23:20:39.306845 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:39.299537    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-479107-6d88b6b77d-hzkkg"
Oct 31 23:20:41.096472 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:41.096430    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85b90bac-ghwkw"]
Oct 31 23:20:41.230167 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:41.230130    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-85b90bac-ghwkw"]
Oct 31 23:20:41.799383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:41.788324    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfd9aa59-6685-4b51-976a-cd4dfe5802b4" path="/var/lib/kubelet/pods/cfd9aa59-6685-4b51-976a-cd4dfe5802b4/volumes"
Oct 31 23:20:42.095420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:42.095338    4301 scope.go:122] "RemoveContainer" containerID="e314a5658651f8fb7824743383a36a3cfe04b9825295619b07e0c2f932a6cc08"
Oct 31 23:20:50.070455 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.070416    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"]
Oct 31 23:20:50.071482 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.071462    4301 cpu_manager.go:414] "containerMap: removing container" podUID="ad66adfb-6f3e-43bf-8499-4b63285c1849" containerName="payload-pre-processing"
Oct 31 23:20:50.071551 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.071486    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad66adfb-6f3e-43bf-8499-4b63285c1849" containerName="payload-pre-processing"
Oct 31 23:20:50.071659 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.071648    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad66adfb-6f3e-43bf-8499-4b63285c1849" containerName="payload-pre-processing"
Oct 31 23:20:50.077680 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.077656    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:20:50.083704 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.083425    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"]
Oct 31 23:20:50.088318 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.088296    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-4eff5f-openshift-default-dockercfg-lfwft\""
Oct 31 23:20:50.224670 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.224635    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pvwq\" (UniqueName: \"kubernetes.io/projected/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-kube-api-access-4pvwq\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:20:50.224670 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.224683    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-credential-socket\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:20:50.224899 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.224864    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istio-envoy\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:20:50.225184 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.225159    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istio-podinfo\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:20:50.225324 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.225216    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istio-token\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:20:50.225324 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.225287    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istio-data\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:20:50.225559 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.225414    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-workload-certs\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:20:50.225559 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.225452    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istiod-ca-cert\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:20:50.225559 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.225502    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-workload-socket\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:20:50.341442 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.335988    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istio-podinfo\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:20:50.355440 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.352842    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istio-token\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:20:50.355440 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.352900    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istio-data\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:20:50.355440 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.352985    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-workload-certs\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:20:50.355440 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.353032    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istiod-ca-cert\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:20:50.355440 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.353104    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-workload-socket\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:20:50.355440 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.353184    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pvwq\" (UniqueName: \"kubernetes.io/projected/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-kube-api-access-4pvwq\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:20:50.355440 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.353219    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-credential-socket\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:20:50.355440 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.353290    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istio-envoy\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:20:50.355440 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.354787    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istiod-ca-cert\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:20:50.355440 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.355348    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istio-data\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:20:50.356145 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.355632    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-workload-certs\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:20:50.400382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.396869    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pvwq\" (UniqueName: \"kubernetes.io/projected/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-kube-api-access-4pvwq\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:20:50.406466 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.403711    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istio-envoy\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:20:50.406683 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.404101    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istio-token\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:20:50.406795 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.404920    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-credential-socket\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:20:50.406894 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.405144    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-workload-socket\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:20:50.417607 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.417559    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istio-podinfo\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:20:50.742382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:50.741868    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:20:51.306793 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:51.306700    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"]
Oct 31 23:20:51.310339 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:20:51.310313    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd05b75f2_18af_4d86_99d9_a51d2d23c3b5.slice/crio-0833dac9bbe70569d8f85a4a7028af2480f46a2526e0f0977c1392cb6be07926 WatchSource:0}: Error finding container 0833dac9bbe70569d8f85a4a7028af2480f46a2526e0f0977c1392cb6be07926: Status 404 returned error can't find the container with id 0833dac9bbe70569d8f85a4a7028af2480f46a2526e0f0977c1392cb6be07926
Oct 31 23:20:51.313330 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:51.313292    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:20:51.313461 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:51.313393    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:20:51.313461 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:51.313442    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:20:51.743529 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:51.743487    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62" event={"ID":"d05b75f2-18af-4d86-99d9-a51d2d23c3b5","Type":"ContainerStarted","Data":"4a30b386bc535202ee7f79b1b6b7bbceaac019d88ca04099ae4c0367bf107eba"}
Oct 31 23:20:51.743737 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:51.743558    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62" event={"ID":"d05b75f2-18af-4d86-99d9-a51d2d23c3b5","Type":"ContainerStarted","Data":"0833dac9bbe70569d8f85a4a7028af2480f46a2526e0f0977c1392cb6be07926"}
Oct 31 23:20:51.770197 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:51.767636    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62" podStartSLOduration=1.767621302 podStartE2EDuration="1.767621302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:20:51.765524097 +0000 UTC m=+1870.512713602" watchObservedRunningTime="2026-10-31 23:20:51.767621302 +0000 UTC m=+1870.514810806"
Oct 31 23:20:52.742716 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:52.742671    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:20:52.752001 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:52.751973    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:20:52.754070 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:52.754035    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:20:52.754825 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:20:52.754803    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:21:02.547762 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:21:02.547728    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:21:04.189168 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:04.189130    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2294a7-xjdj2"]
Oct 31 23:21:04.195438 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:04.195414    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2294a7-xjdj2"
Oct 31 23:21:04.198825 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:04.198800    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 23:21:04.198986 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:04.198967    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\""
Oct 31 23:21:04.202148 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:04.202126    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2294a7-xjdj2"]
Oct 31 23:21:04.305488 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:04.305453    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bc5s\" (UniqueName: \"kubernetes.io/projected/08697f56-4d6e-4660-8986-13bd439d2f16-kube-api-access-6bc5s\") pod \"maas-api-revoke-keys-e2e-worker-w5-2294a7-xjdj2\" (UID: \"08697f56-4d6e-4660-8986-13bd439d2f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2294a7-xjdj2"
Oct 31 23:21:04.305717 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:04.305673    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08697f56-4d6e-4660-8986-13bd439d2f16-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2294a7-xjdj2\" (UID: \"08697f56-4d6e-4660-8986-13bd439d2f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2294a7-xjdj2"
Oct 31 23:21:04.305788 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:04.305722    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08697f56-4d6e-4660-8986-13bd439d2f16-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2294a7-xjdj2\" (UID: \"08697f56-4d6e-4660-8986-13bd439d2f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2294a7-xjdj2"
Oct 31 23:21:04.407348 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:04.407315    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bc5s\" (UniqueName: \"kubernetes.io/projected/08697f56-4d6e-4660-8986-13bd439d2f16-kube-api-access-6bc5s\") pod \"maas-api-revoke-keys-e2e-worker-w5-2294a7-xjdj2\" (UID: \"08697f56-4d6e-4660-8986-13bd439d2f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2294a7-xjdj2"
Oct 31 23:21:04.407533 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:04.407478    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08697f56-4d6e-4660-8986-13bd439d2f16-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2294a7-xjdj2\" (UID: \"08697f56-4d6e-4660-8986-13bd439d2f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2294a7-xjdj2"
Oct 31 23:21:04.407533 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:04.407521    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08697f56-4d6e-4660-8986-13bd439d2f16-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2294a7-xjdj2\" (UID: \"08697f56-4d6e-4660-8986-13bd439d2f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2294a7-xjdj2"
Oct 31 23:21:04.408247 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:04.408223    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08697f56-4d6e-4660-8986-13bd439d2f16-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2294a7-xjdj2\" (UID: \"08697f56-4d6e-4660-8986-13bd439d2f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2294a7-xjdj2"
Oct 31 23:21:04.409954 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:04.409925    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08697f56-4d6e-4660-8986-13bd439d2f16-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2294a7-xjdj2\" (UID: \"08697f56-4d6e-4660-8986-13bd439d2f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2294a7-xjdj2"
Oct 31 23:21:04.418855 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:04.418832    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bc5s\" (UniqueName: \"kubernetes.io/projected/08697f56-4d6e-4660-8986-13bd439d2f16-kube-api-access-6bc5s\") pod \"maas-api-revoke-keys-e2e-worker-w5-2294a7-xjdj2\" (UID: \"08697f56-4d6e-4660-8986-13bd439d2f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2294a7-xjdj2"
Oct 31 23:21:04.514973 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:04.514901    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2294a7-xjdj2"
Oct 31 23:21:04.799159 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:04.799094    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2294a7-xjdj2"]
Oct 31 23:21:04.905595 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:04.905557    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2294a7-xjdj2" event={"ID":"08697f56-4d6e-4660-8986-13bd439d2f16","Type":"ContainerStarted","Data":"3d7f0177927646a7bc8d23b383a67603285b32d19b1e8c138592695ad181a0fd"}
Oct 31 23:21:05.914066 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:05.914032    4301 generic.go:356] "Generic (PLEG): container finished" podID="08697f56-4d6e-4660-8986-13bd439d2f16" containerID="e8f35cbd467eb603ddcac5b5c41fa555104ac1c32843b6a1d6ccd98a242d8c06" exitCode=0
Oct 31 23:21:05.914554 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:05.914164    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2294a7-xjdj2" event={"ID":"08697f56-4d6e-4660-8986-13bd439d2f16","Type":"ContainerDied","Data":"e8f35cbd467eb603ddcac5b5c41fa555104ac1c32843b6a1d6ccd98a242d8c06"}
Oct 31 23:21:07.122106 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:07.122081    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2294a7-xjdj2"
Oct 31 23:21:07.185719 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:07.185690    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08697f56-4d6e-4660-8986-13bd439d2f16-kube-api-access-6bc5s\" (UniqueName: \"kubernetes.io/projected/08697f56-4d6e-4660-8986-13bd439d2f16-kube-api-access-6bc5s\") pod \"08697f56-4d6e-4660-8986-13bd439d2f16\" (UID: \"08697f56-4d6e-4660-8986-13bd439d2f16\") "
Oct 31 23:21:07.185988 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:07.185968    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08697f56-4d6e-4660-8986-13bd439d2f16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08697f56-4d6e-4660-8986-13bd439d2f16-maas-api-xks-ca\") pod \"08697f56-4d6e-4660-8986-13bd439d2f16\" (UID: \"08697f56-4d6e-4660-8986-13bd439d2f16\") "
Oct 31 23:21:07.186125 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:07.186104    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08697f56-4d6e-4660-8986-13bd439d2f16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08697f56-4d6e-4660-8986-13bd439d2f16-maas-api-openshift-ca\") pod \"08697f56-4d6e-4660-8986-13bd439d2f16\" (UID: \"08697f56-4d6e-4660-8986-13bd439d2f16\") "
Oct 31 23:21:07.186495 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:07.186469    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08697f56-4d6e-4660-8986-13bd439d2f16-maas-api-openshift-ca" pod "08697f56-4d6e-4660-8986-13bd439d2f16" (UID: "08697f56-4d6e-4660-8986-13bd439d2f16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:21:07.186786 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:07.186768    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08697f56-4d6e-4660-8986-13bd439d2f16-maas-api-openshift-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:21:07.188410 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:07.188327    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08697f56-4d6e-4660-8986-13bd439d2f16-maas-api-xks-ca" pod "08697f56-4d6e-4660-8986-13bd439d2f16" (UID: "08697f56-4d6e-4660-8986-13bd439d2f16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:21:07.188496 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:07.188336    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08697f56-4d6e-4660-8986-13bd439d2f16-kube-api-access-6bc5s" pod "08697f56-4d6e-4660-8986-13bd439d2f16" (UID: "08697f56-4d6e-4660-8986-13bd439d2f16"). InnerVolumeSpecName "kube-api-access-6bc5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:21:07.287881 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:07.287853    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08697f56-4d6e-4660-8986-13bd439d2f16-maas-api-xks-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:21:07.287881 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:07.287878    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bc5s\" (UniqueName: \"kubernetes.io/projected/08697f56-4d6e-4660-8986-13bd439d2f16-kube-api-access-6bc5s\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:21:07.953129 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:07.953095    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2294a7-xjdj2"
Oct 31 23:21:07.953310 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:07.953097    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2294a7-xjdj2" event={"ID":"08697f56-4d6e-4660-8986-13bd439d2f16","Type":"ContainerDied","Data":"3d7f0177927646a7bc8d23b383a67603285b32d19b1e8c138592695ad181a0fd"}
Oct 31 23:21:07.953310 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:07.953219    4301 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d7f0177927646a7bc8d23b383a67603285b32d19b1e8c138592695ad181a0fd"
Oct 31 23:21:11.553600 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.553559    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"]
Oct 31 23:21:11.579467 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.579432    4301 cpu_manager.go:414] "containerMap: removing container" podUID="08697f56-4d6e-4660-8986-13bd439d2f16" containerName="revoke-keys"
Oct 31 23:21:11.595446 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.595404    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08697f56-4d6e-4660-8986-13bd439d2f16" containerName="revoke-keys"
Oct 31 23:21:11.595977 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.595948    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="08697f56-4d6e-4660-8986-13bd439d2f16" containerName="revoke-keys"
Oct 31 23:21:11.600673 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.600647    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"]
Oct 31 23:21:11.600965 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.600947    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:21:11.623108 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.622848    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-885fdf-openshift-default-dockercfg-9mwfb\""
Oct 31 23:21:11.763383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.736851    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istio-podinfo\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:21:11.766231 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.766199    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istio-token\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:21:11.766532 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.766510    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-credential-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:21:11.766687 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.766673    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-workload-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:21:11.766791 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.766775    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-workload-certs\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:21:11.777383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.771132    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istiod-ca-cert\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:21:11.777879 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.777845    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istio-data\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:21:11.795870 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.795827    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istio-envoy\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:21:11.796248 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.796223    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctk55\" (UniqueName: \"kubernetes.io/projected/46881476-3103-4ee9-9b90-ec0ba9f30c3e-kube-api-access-ctk55\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:21:11.913292 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.913249    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istio-data\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:21:11.915258 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.915187    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istio-envoy\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:21:11.915738 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.915699    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctk55\" (UniqueName: \"kubernetes.io/projected/46881476-3103-4ee9-9b90-ec0ba9f30c3e-kube-api-access-ctk55\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:21:11.917126 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.917085    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istio-podinfo\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:21:11.917401 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.917379    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istio-token\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:21:11.917541 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.917522    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-credential-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:21:11.917735 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.917719    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-workload-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:21:11.917869 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.917840    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-workload-certs\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:21:11.918101 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.918071    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-credential-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:21:11.942446 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.918164    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istiod-ca-cert\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:21:11.942880 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.918882    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istiod-ca-cert\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:21:11.943019 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.919132    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-workload-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:21:12.020643 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.914802    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istio-data\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:21:12.020643 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.919376    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-workload-certs\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:21:12.020643 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.925846    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istio-podinfo\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:21:12.020643 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.938229    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istio-envoy\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:21:12.020643 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.954034    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istio-token\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:21:12.020643 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:11.957109    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctk55\" (UniqueName: \"kubernetes.io/projected/46881476-3103-4ee9-9b90-ec0ba9f30c3e-kube-api-access-ctk55\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:21:12.090007 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:12.073706    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:21:12.565000 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:21:12.564968    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46881476_3103_4ee9_9b90_ec0ba9f30c3e.slice/crio-2186d97fd07165f3bd0d10061bc6afb43157b53884132faf92107d64623f2ed9 WatchSource:0}: Error finding container 2186d97fd07165f3bd0d10061bc6afb43157b53884132faf92107d64623f2ed9: Status 404 returned error can't find the container with id 2186d97fd07165f3bd0d10061bc6afb43157b53884132faf92107d64623f2ed9
Oct 31 23:21:12.566994 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:12.566878    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"]
Oct 31 23:21:12.568686 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:12.568643    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:21:12.568773 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:12.568751    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:21:12.568818 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:12.568794    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:21:13.243127 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:13.243085    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7" event={"ID":"46881476-3103-4ee9-9b90-ec0ba9f30c3e","Type":"ContainerStarted","Data":"876cb199f3051d7a7d6abea4aba1b81d36582ea4122d5e6e269e4ca44106574d"}
Oct 31 23:21:13.243339 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:13.243180    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7" event={"ID":"46881476-3103-4ee9-9b90-ec0ba9f30c3e","Type":"ContainerStarted","Data":"2186d97fd07165f3bd0d10061bc6afb43157b53884132faf92107d64623f2ed9"}
Oct 31 23:21:13.356383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:13.353995    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7" podStartSLOduration=2.353973673 podStartE2EDuration="2.353973673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:21:13.329057303 +0000 UTC m=+1892.076246810" watchObservedRunningTime="2026-10-31 23:21:13.353973673 +0000 UTC m=+1892.101163204"
Oct 31 23:21:14.085601 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:14.085569    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:21:14.092739 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:14.092707    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:21:14.414942 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:14.414913    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:21:14.421384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:14.421335    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:21:32.634986 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:21:32.634946    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:21:39.479750 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:39.479712    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"]
Oct 31 23:21:39.480271 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:39.479937    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk" podUID="71069228-75f3-453b-92b7-2648a3921d39" containerName="istio-proxy" containerID="cri-o://ae5c3f76dc49255504c1d601a5010fcfb01fa885c32cbe2f78f3eb1192d38c27" gracePeriod=30
Oct 31 23:21:45.099585 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:45.099485    4301 generic.go:356] "Generic (PLEG): container finished" podID="71069228-75f3-453b-92b7-2648a3921d39" containerID="ae5c3f76dc49255504c1d601a5010fcfb01fa885c32cbe2f78f3eb1192d38c27" exitCode=0
Oct 31 23:21:45.100381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:45.099571    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk" event={"ID":"71069228-75f3-453b-92b7-2648a3921d39","Type":"ContainerDied","Data":"ae5c3f76dc49255504c1d601a5010fcfb01fa885c32cbe2f78f3eb1192d38c27"}
Oct 31 23:21:45.191680 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:45.191656    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:21:45.258526 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:45.258486    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71069228-75f3-453b-92b7-2648a3921d39-kube-api-access-b5zt2\" (UniqueName: \"kubernetes.io/projected/71069228-75f3-453b-92b7-2648a3921d39-kube-api-access-b5zt2\") pod \"71069228-75f3-453b-92b7-2648a3921d39\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") "
Oct 31 23:21:45.258696 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:45.258583    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-istio-envoy\") pod \"71069228-75f3-453b-92b7-2648a3921d39\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") "
Oct 31 23:21:45.258696 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:45.258635    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-istio-data\") pod \"71069228-75f3-453b-92b7-2648a3921d39\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") "
Oct 31 23:21:45.258696 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:45.258679    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-credential-socket\") pod \"71069228-75f3-453b-92b7-2648a3921d39\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") "
Oct 31 23:21:45.258874 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:45.258750    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-workload-certs\") pod \"71069228-75f3-453b-92b7-2648a3921d39\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") "
Oct 31 23:21:45.258985 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:45.258964    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-workload-socket\") pod \"71069228-75f3-453b-92b7-2648a3921d39\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") "
Oct 31 23:21:45.259057 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:45.259013    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71069228-75f3-453b-92b7-2648a3921d39-istio-token\" (UniqueName: \"kubernetes.io/projected/71069228-75f3-453b-92b7-2648a3921d39-istio-token\") pod \"71069228-75f3-453b-92b7-2648a3921d39\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") "
Oct 31 23:21:45.259177 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:45.259157    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71069228-75f3-453b-92b7-2648a3921d39-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71069228-75f3-453b-92b7-2648a3921d39-istiod-ca-cert\") pod \"71069228-75f3-453b-92b7-2648a3921d39\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") "
Oct 31 23:21:45.259392 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:45.259210    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-workload-certs" pod "71069228-75f3-453b-92b7-2648a3921d39" (UID: "71069228-75f3-453b-92b7-2648a3921d39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:21:45.259487 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:45.259228    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-credential-socket" pod "71069228-75f3-453b-92b7-2648a3921d39" (UID: "71069228-75f3-453b-92b7-2648a3921d39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:21:45.259487 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:45.259328    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-workload-socket" pod "71069228-75f3-453b-92b7-2648a3921d39" (UID: "71069228-75f3-453b-92b7-2648a3921d39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:21:45.259600 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:45.259564    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71069228-75f3-453b-92b7-2648a3921d39-istiod-ca-cert" pod "71069228-75f3-453b-92b7-2648a3921d39" (UID: "71069228-75f3-453b-92b7-2648a3921d39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:21:45.259780 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:45.259760    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71069228-75f3-453b-92b7-2648a3921d39-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71069228-75f3-453b-92b7-2648a3921d39-istio-podinfo\") pod \"71069228-75f3-453b-92b7-2648a3921d39\" (UID: \"71069228-75f3-453b-92b7-2648a3921d39\") "
Oct 31 23:21:45.260065 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:45.259819    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-istio-data" pod "71069228-75f3-453b-92b7-2648a3921d39" (UID: "71069228-75f3-453b-92b7-2648a3921d39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:21:45.260749 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:45.260720    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71069228-75f3-453b-92b7-2648a3921d39-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:21:45.260923 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:45.260755    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:21:45.260923 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:45.260771    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:21:45.260923 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:45.260785    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:21:45.260923 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:45.260799    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:21:45.261459 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:45.261432    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71069228-75f3-453b-92b7-2648a3921d39-kube-api-access-b5zt2" pod "71069228-75f3-453b-92b7-2648a3921d39" (UID: "71069228-75f3-453b-92b7-2648a3921d39"). InnerVolumeSpecName "kube-api-access-b5zt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:21:45.261694 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:45.261668    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71069228-75f3-453b-92b7-2648a3921d39-istio-token" pod "71069228-75f3-453b-92b7-2648a3921d39" (UID: "71069228-75f3-453b-92b7-2648a3921d39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:21:45.262687 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:45.262667    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-istio-envoy" pod "71069228-75f3-453b-92b7-2648a3921d39" (UID: "71069228-75f3-453b-92b7-2648a3921d39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:21:45.262994 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:45.262976    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71069228-75f3-453b-92b7-2648a3921d39-istio-podinfo" pod "71069228-75f3-453b-92b7-2648a3921d39" (UID: "71069228-75f3-453b-92b7-2648a3921d39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:21:45.384301 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:45.378563    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5zt2\" (UniqueName: \"kubernetes.io/projected/71069228-75f3-453b-92b7-2648a3921d39-kube-api-access-b5zt2\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:21:45.384301 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:45.378601    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71069228-75f3-453b-92b7-2648a3921d39-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:21:45.384301 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:45.378616    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71069228-75f3-453b-92b7-2648a3921d39-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:21:45.384301 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:45.378631    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71069228-75f3-453b-92b7-2648a3921d39-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:21:46.109981 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:46.109947    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"
Oct 31 23:21:46.109981 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:46.109974    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk" event={"ID":"71069228-75f3-453b-92b7-2648a3921d39","Type":"ContainerDied","Data":"0335f0b13d441e29ffad3a6ab074f9c63f3db2270deb955536b31fac461f9e34"}
Oct 31 23:21:46.110539 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:46.110036    4301 scope.go:122] "RemoveContainer" containerID="ae5c3f76dc49255504c1d601a5010fcfb01fa885c32cbe2f78f3eb1192d38c27"
Oct 31 23:21:46.281531 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:46.280717    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"]
Oct 31 23:21:46.285178 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:46.285148    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-sn8wk"]
Oct 31 23:21:47.684413 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:47.684334    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71069228-75f3-453b-92b7-2648a3921d39" path="/var/lib/kubelet/pods/71069228-75f3-453b-92b7-2648a3921d39/volumes"
Oct 31 23:21:50.902451 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:50.902413    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2294a7-xjdj2"]
Oct 31 23:21:50.941585 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:50.941551    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2294a7-xjdj2"]
Oct 31 23:21:51.475402 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:21:51.475346    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08697f56-4d6e-4660-8986-13bd439d2f16" path="/var/lib/kubelet/pods/08697f56-4d6e-4660-8986-13bd439d2f16/volumes"
Oct 31 23:22:01.937019 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:01.936920    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"]
Oct 31 23:22:01.937517 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:01.937175    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs" podUID="944451b3-0e62-4419-bf3a-37401268d08f" containerName="istio-proxy" containerID="cri-o://fb09bceafe3fe9ee13ce54940d702896cd7be3b03819ead1f800208d4b19f688" gracePeriod=30
Oct 31 23:22:01.943536 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:01.943504    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"]
Oct 31 23:22:01.943736 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:01.943715    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2" podUID="f421d513-57d6-43b1-af00-bed486a4d0bf" containerName="istio-proxy" containerID="cri-o://26c66833d17c4de159bed0375522a142211248db1eb912d9c40bbce7f7f97004" gracePeriod=30
Oct 31 23:22:01.952393 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:01.949961    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"]
Oct 31 23:22:01.952393 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:01.950195    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm" podUID="de434cd6-3610-47ac-ae7e-78e1d05cb0c5" containerName="istio-proxy" containerID="cri-o://8342fb6cfa9189bbefe86a43bbe1878ce7caae656611eeb74aac34a0ee189a9c" gracePeriod=30
Oct 31 23:22:02.116388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:02.114524    4301 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 23:22:02.116388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:02.114585    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm" podUID="de434cd6-3610-47ac-ae7e-78e1d05cb0c5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 23:22:02.743408 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:22:02.743319    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:22:03.714556 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:03.714513    4301 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 23:22:03.715157 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:03.715125    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2" podUID="f421d513-57d6-43b1-af00-bed486a4d0bf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 23:22:04.117674 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:04.117557    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:04.239970 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:04.239926    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:04.351597 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:04.351535    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:04.410934 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:04.400443    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97" podUID="c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:04.439397 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:04.438485    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 23:22:04.516390 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:04.516028    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:05.013521 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:05.013470    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:05.028602 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:05.028557    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:05.123311 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:05.120482    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:05.123311 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:05.121504    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:22:05.266871 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:05.266770    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:05.281309 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:05.267773    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:22:05.353553 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:05.353502    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:05.412429 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:05.356556    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:22:05.414053 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:05.414017    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97" podUID="c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:05.472391 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:05.420938    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"
Oct 31 23:22:05.472391 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:05.443027    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:05.472391 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:05.443237    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 31 23:22:05.505387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:05.494623    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:05.569607 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:05.569469    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:05.570171 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:05.570142    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 23:22:05.976397 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:05.943279    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:22:05.977147 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:05.977122    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:22:05.977339 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:05.977328    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:22:06.103408 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:06.103323    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:06.104795 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:06.104769    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:22:06.134554 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:06.105439    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:06.134554 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:06.113812    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:22:06.313609 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:06.313504    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:06.466230 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:06.459500    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 31 23:22:06.593514 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:06.532878    4301 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 23:22:06.593514 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:06.532950    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs" podUID="944451b3-0e62-4419-bf3a-37401268d08f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 23:22:06.640613 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:06.640565    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:06.665699 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:06.665655    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 23:22:06.667991 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:06.666741    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:22:06.714401 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:06.700688    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97" podUID="c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:06.881511 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:06.881474    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 23:22:06.965402 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:06.925406    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded"
Oct 31 23:22:07.221660 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:07.146660    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 31 23:22:07.221660 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:07.158306    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 23:22:07.221660 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:07.158772    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:07.368951 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:07.353206    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 23:22:07.370295 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:07.369958    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:07.370295 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:07.370140    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded"
Oct 31 23:22:07.668793 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:07.628123    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 31 23:22:08.004515 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:08.004393    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": context deadline exceeded"
Oct 31 23:22:08.045431 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:08.032702    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 23:22:08.118697 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:08.118655    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 31 23:22:08.200481 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:08.152889    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:08.200481 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:08.160799    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:08.200481 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:08.161540    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 23:22:08.243428 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:08.242301    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97" podUID="c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 31 23:22:08.273311 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:08.272731    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:08.277640 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:08.277590    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 23:22:08.384135 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:08.382556    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:08.414479 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:08.298986    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 31 23:22:08.675607 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:08.661619    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:08.842854 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:22:08.842640    4301 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.445s"
Oct 31 23:22:08.847828 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:08.847628    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:22:08.910598 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:08.910555    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94ecade3-w8zp7"]
Oct 31 23:22:08.937169 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:08.937056    4301 cpu_manager.go:414] "containerMap: removing container" podUID="71069228-75f3-453b-92b7-2648a3921d39" containerName="istio-proxy"
Oct 31 23:22:08.937412 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:08.937392    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71069228-75f3-453b-92b7-2648a3921d39" containerName="istio-proxy"
Oct 31 23:22:08.937807 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:08.937791    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="71069228-75f3-453b-92b7-2648a3921d39" containerName="istio-proxy"
Oct 31 23:22:09.045754 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:09.045698    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:09.067578 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:09.053393    4301 generic.go:356] "Generic (PLEG): container finished" podID="f421d513-57d6-43b1-af00-bed486a4d0bf" containerID="26c66833d17c4de159bed0375522a142211248db1eb912d9c40bbce7f7f97004" exitCode=0
Oct 31 23:22:09.098720 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:09.062128    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2" event={"ID":"f421d513-57d6-43b1-af00-bed486a4d0bf","Type":"ContainerDied","Data":"26c66833d17c4de159bed0375522a142211248db1eb912d9c40bbce7f7f97004"}
Oct 31 23:22:09.099196 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:09.062231    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ecade3-w8zp7"
Oct 31 23:22:09.126135 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:22:09.101712    4301 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde434cd6_3610_47ac_ae7e_78e1d05cb0c5.slice/crio-8342fb6cfa9189bbefe86a43bbe1878ce7caae656611eeb74aac34a0ee189a9c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf421d513_57d6_43b1_af00_bed486a4d0bf.slice/crio-26c66833d17c4de159bed0375522a142211248db1eb912d9c40bbce7f7f97004.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf421d513_57d6_43b1_af00_bed486a4d0bf.slice/crio-conmon-26c66833d17c4de159bed0375522a142211248db1eb912d9c40bbce7f7f97004.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944451b3_0e62_4419_bf3a_37401268d08f.slice/crio-fb09bceafe3fe9ee13ce54940d702896cd7be3b03819ead1f800208d4b19f688.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944451b3_0e62_4419_bf3a_37401268d08f.slice/crio-conmon-fb09bceafe3fe9ee13ce54940d702896cd7be3b03819ead1f800208d4b19f688.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde434cd6_3610_47ac_ae7e_78e1d05cb0c5.slice/crio-conmon-8342fb6cfa9189bbefe86a43bbe1878ce7caae656611eeb74aac34a0ee189a9c.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 23:22:09.451472 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:22:09.070676    4301 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde434cd6_3610_47ac_ae7e_78e1d05cb0c5.slice/crio-conmon-8342fb6cfa9189bbefe86a43bbe1878ce7caae656611eeb74aac34a0ee189a9c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf421d513_57d6_43b1_af00_bed486a4d0bf.slice/crio-conmon-26c66833d17c4de159bed0375522a142211248db1eb912d9c40bbce7f7f97004.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944451b3_0e62_4419_bf3a_37401268d08f.slice/crio-conmon-fb09bceafe3fe9ee13ce54940d702896cd7be3b03819ead1f800208d4b19f688.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf421d513_57d6_43b1_af00_bed486a4d0bf.slice/crio-26c66833d17c4de159bed0375522a142211248db1eb912d9c40bbce7f7f97004.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944451b3_0e62_4419_bf3a_37401268d08f.slice/crio-fb09bceafe3fe9ee13ce54940d702896cd7be3b03819ead1f800208d4b19f688.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 23:22:09.455518 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:09.455479    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 23:22:09.474541 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:09.168754    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 23:22:09.479498 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:09.168791    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 23:22:09.479932 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:09.181484    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 31 23:22:09.480524 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:09.183171    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 23:22:09.480676 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:09.183585    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\""
Oct 31 23:22:09.480768 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:09.219746    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94ecade3-w8zp7"]
Oct 31 23:22:09.480945 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:09.480930    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:22:09.514385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:09.501560    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:09.514385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:09.458214    4301 generic.go:356] "Generic (PLEG): container finished" podID="de434cd6-3610-47ac-ae7e-78e1d05cb0c5" containerID="8342fb6cfa9189bbefe86a43bbe1878ce7caae656611eeb74aac34a0ee189a9c" exitCode=0
Oct 31 23:22:09.514385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:09.120994    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:09.514385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:09.273505    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97" podUID="c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:09.514385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:09.349267    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05c2fd23-331c-4818-b14c-786d6e14477a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94ecade3-w8zp7\" (UID: \"05c2fd23-331c-4818-b14c-786d6e14477a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ecade3-w8zp7"
Oct 31 23:22:09.524516 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:09.524461    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05c2fd23-331c-4818-b14c-786d6e14477a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94ecade3-w8zp7\" (UID: \"05c2fd23-331c-4818-b14c-786d6e14477a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ecade3-w8zp7"
Oct 31 23:22:09.525083 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:09.525060    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65mc2\" (UniqueName: \"kubernetes.io/projected/05c2fd23-331c-4818-b14c-786d6e14477a-kube-api-access-65mc2\") pod \"maas-api-revoke-sub-94ecade3-w8zp7\" (UID: \"05c2fd23-331c-4818-b14c-786d6e14477a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ecade3-w8zp7"
Oct 31 23:22:09.695385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:09.634216    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05c2fd23-331c-4818-b14c-786d6e14477a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94ecade3-w8zp7\" (UID: \"05c2fd23-331c-4818-b14c-786d6e14477a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ecade3-w8zp7"
Oct 31 23:22:09.753910 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:09.753815    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05c2fd23-331c-4818-b14c-786d6e14477a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94ecade3-w8zp7\" (UID: \"05c2fd23-331c-4818-b14c-786d6e14477a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ecade3-w8zp7"
Oct 31 23:22:09.786395 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:09.786329    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65mc2\" (UniqueName: \"kubernetes.io/projected/05c2fd23-331c-4818-b14c-786d6e14477a-kube-api-access-65mc2\") pod \"maas-api-revoke-sub-94ecade3-w8zp7\" (UID: \"05c2fd23-331c-4818-b14c-786d6e14477a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ecade3-w8zp7"
Oct 31 23:22:09.787019 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:09.741158    4301 generic.go:356] "Generic (PLEG): container finished" podID="944451b3-0e62-4419-bf3a-37401268d08f" containerID="fb09bceafe3fe9ee13ce54940d702896cd7be3b03819ead1f800208d4b19f688" exitCode=0
Oct 31 23:22:09.800845 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:09.687238    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 31 23:22:09.840386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:09.773056    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm" event={"ID":"de434cd6-3610-47ac-ae7e-78e1d05cb0c5","Type":"ContainerDied","Data":"8342fb6cfa9189bbefe86a43bbe1878ce7caae656611eeb74aac34a0ee189a9c"}
Oct 31 23:22:09.840744 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:09.840708    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs" event={"ID":"944451b3-0e62-4419-bf3a-37401268d08f","Type":"ContainerDied","Data":"fb09bceafe3fe9ee13ce54940d702896cd7be3b03819ead1f800208d4b19f688"}
Oct 31 23:22:09.840961 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:09.711588    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05c2fd23-331c-4818-b14c-786d6e14477a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94ecade3-w8zp7\" (UID: \"05c2fd23-331c-4818-b14c-786d6e14477a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ecade3-w8zp7"
Oct 31 23:22:09.841180 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:09.841163    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:22:09.898724 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:09.898684    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65mc2\" (UniqueName: \"kubernetes.io/projected/05c2fd23-331c-4818-b14c-786d6e14477a-kube-api-access-65mc2\") pod \"maas-api-revoke-sub-94ecade3-w8zp7\" (UID: \"05c2fd23-331c-4818-b14c-786d6e14477a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ecade3-w8zp7"
Oct 31 23:22:10.071550 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:10.057909    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05c2fd23-331c-4818-b14c-786d6e14477a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94ecade3-w8zp7\" (UID: \"05c2fd23-331c-4818-b14c-786d6e14477a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ecade3-w8zp7"
Oct 31 23:22:10.072019 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:10.071986    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ecade3-w8zp7"
Oct 31 23:22:10.165519 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:10.142155    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 23:22:10.424901 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:10.424859    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:10.477792 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:10.477746    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:10.479947 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:10.479914    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:10.480639 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:10.480613    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 23:22:10.481186 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:10.481160    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:10.503923 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:10.501830    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:10.506317 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:10.505509    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97" podUID="c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:10.508076 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:10.506562    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:10.541077 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:10.541049    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:22:10.626500 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:10.626455    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded"
Oct 31 23:22:10.810421 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:10.810335    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:11.325154 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:11.323905    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:11.325154 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:11.324151    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": context deadline exceeded"
Oct 31 23:22:11.530422 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:11.524525    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:11.530422 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:11.524994    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97" podUID="c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:11.541586 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:11.531566    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:11.563098 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:11.551960    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:11.684497 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:11.666415    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 23:22:11.698786 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:11.698504    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:11.727384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:11.708533    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:11.970544 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:11.836194    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:22:11.970544 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:11.836476    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:22:12.313178 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:12.290462    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97" podUID="c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:12.313178 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:12.291350    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:22:12.353110 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:12.316667    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:22:12.568457 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:12.353299    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:12.830435 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:12.803409    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:13.011539 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:13.011489    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:13.045831 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:13.045791    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:22:13.181415 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:13.139762    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 23:22:13.207389 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:13.186974    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 31 23:22:13.392483 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:13.348903    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97" podUID="c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:13.652333 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:13.444296    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:22:13.771618 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:13.771575    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:13.804959 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:13.791247    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:13.861169 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:13.861123    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f421d513-57d6-43b1-af00-bed486a4d0bf-istio-token\" (UniqueName: \"kubernetes.io/projected/f421d513-57d6-43b1-af00-bed486a4d0bf-istio-token\") pod \"f421d513-57d6-43b1-af00-bed486a4d0bf\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") "
Oct 31 23:22:13.861593 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:13.861566    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f421d513-57d6-43b1-af00-bed486a4d0bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f421d513-57d6-43b1-af00-bed486a4d0bf-istio-podinfo\") pod \"f421d513-57d6-43b1-af00-bed486a4d0bf\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") "
Oct 31 23:22:13.862292 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:13.862270    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-workload-certs\") pod \"f421d513-57d6-43b1-af00-bed486a4d0bf\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") "
Oct 31 23:22:14.173835 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.173799    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f421d513-57d6-43b1-af00-bed486a4d0bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f421d513-57d6-43b1-af00-bed486a4d0bf-istiod-ca-cert\") pod \"f421d513-57d6-43b1-af00-bed486a4d0bf\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") "
Oct 31 23:22:14.174194 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.174135    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f421d513-57d6-43b1-af00-bed486a4d0bf-kube-api-access-wscr6\" (UniqueName: \"kubernetes.io/projected/f421d513-57d6-43b1-af00-bed486a4d0bf-kube-api-access-wscr6\") pod \"f421d513-57d6-43b1-af00-bed486a4d0bf\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") "
Oct 31 23:22:14.200449 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.182136    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:14.259420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.257781    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:14.259420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.258299    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:22:14.319752 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.319690    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-workload-certs" pod "f421d513-57d6-43b1-af00-bed486a4d0bf" (UID: "f421d513-57d6-43b1-af00-bed486a4d0bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:14.367769 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.367722    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97" podUID="c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:14.384468 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.369314    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 31 23:22:14.480732 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.480687    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:22:14.552764 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.482216    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:22:14.643386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.629565    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-workload-socket\") pod \"f421d513-57d6-43b1-af00-bed486a4d0bf\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") "
Oct 31 23:22:14.643809 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.643763    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/944451b3-0e62-4419-bf3a-37401268d08f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/944451b3-0e62-4419-bf3a-37401268d08f-istio-podinfo\") pod \"944451b3-0e62-4419-bf3a-37401268d08f\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") "
Oct 31 23:22:14.644001 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.643983    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-credential-socket\") pod \"f421d513-57d6-43b1-af00-bed486a4d0bf\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") "
Oct 31 23:22:14.644231 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.644213    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-kube-api-access-kzfqk\" (UniqueName: \"kubernetes.io/projected/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-kube-api-access-kzfqk\") pod \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") "
Oct 31 23:22:14.644443 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.644423    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/944451b3-0e62-4419-bf3a-37401268d08f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/944451b3-0e62-4419-bf3a-37401268d08f-istiod-ca-cert\") pod \"944451b3-0e62-4419-bf3a-37401268d08f\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") "
Oct 31 23:22:14.644700 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.644682    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-istio-envoy\") pod \"f421d513-57d6-43b1-af00-bed486a4d0bf\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") "
Oct 31 23:22:14.644907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.644889    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-credential-socket\") pod \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") "
Oct 31 23:22:14.645071 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.645054    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/944451b3-0e62-4419-bf3a-37401268d08f-kube-api-access-2dfl9\" (UniqueName: \"kubernetes.io/projected/944451b3-0e62-4419-bf3a-37401268d08f-kube-api-access-2dfl9\") pod \"944451b3-0e62-4419-bf3a-37401268d08f\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") "
Oct 31 23:22:14.650390 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.648572    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istiod-ca-cert\") pod \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") "
Oct 31 23:22:14.650666 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.650644    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istio-data\") pod \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") "
Oct 31 23:22:14.696225 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.667695    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-istio-data\") pod \"944451b3-0e62-4419-bf3a-37401268d08f\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") "
Oct 31 23:22:14.696225 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.667771    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-istio-data\") pod \"f421d513-57d6-43b1-af00-bed486a4d0bf\" (UID: \"f421d513-57d6-43b1-af00-bed486a4d0bf\") "
Oct 31 23:22:14.696225 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.667823    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/944451b3-0e62-4419-bf3a-37401268d08f-istio-token\" (UniqueName: \"kubernetes.io/projected/944451b3-0e62-4419-bf3a-37401268d08f-istio-token\") pod \"944451b3-0e62-4419-bf3a-37401268d08f\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") "
Oct 31 23:22:14.696225 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.669531    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:14.696225 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.659273    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 23:22:14.696225 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.659469    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istio-data" pod "de434cd6-3610-47ac-ae7e-78e1d05cb0c5" (UID: "de434cd6-3610-47ac-ae7e-78e1d05cb0c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:14.696225 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.660743    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f421d513-57d6-43b1-af00-bed486a4d0bf-istiod-ca-cert" pod "f421d513-57d6-43b1-af00-bed486a4d0bf" (UID: "f421d513-57d6-43b1-af00-bed486a4d0bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:22:14.696225 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.664468    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-workload-socket" pod "f421d513-57d6-43b1-af00-bed486a4d0bf" (UID: "f421d513-57d6-43b1-af00-bed486a4d0bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:14.696225 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.695546    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-istio-data" pod "944451b3-0e62-4419-bf3a-37401268d08f" (UID: "944451b3-0e62-4419-bf3a-37401268d08f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:14.696225 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.695874    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-istio-data" pod "f421d513-57d6-43b1-af00-bed486a4d0bf" (UID: "f421d513-57d6-43b1-af00-bed486a4d0bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:14.733070 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.732632    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-credential-socket" pod "f421d513-57d6-43b1-af00-bed486a4d0bf" (UID: "f421d513-57d6-43b1-af00-bed486a4d0bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:14.751320 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.750928    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/944451b3-0e62-4419-bf3a-37401268d08f-istiod-ca-cert" pod "944451b3-0e62-4419-bf3a-37401268d08f" (UID: "944451b3-0e62-4419-bf3a-37401268d08f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:22:14.791926 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.791484    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-credential-socket" pod "de434cd6-3610-47ac-ae7e-78e1d05cb0c5" (UID: "de434cd6-3610-47ac-ae7e-78e1d05cb0c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:14.797001 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.796946    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:14.848644 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.835973    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-workload-certs\") pod \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") "
Oct 31 23:22:14.848644 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.836023    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-credential-socket\") pod \"944451b3-0e62-4419-bf3a-37401268d08f\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") "
Oct 31 23:22:14.848644 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.836124    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-workload-socket\") pod \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") "
Oct 31 23:22:14.848644 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.836168    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istio-podinfo\") pod \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") "
Oct 31 23:22:14.848644 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.836229    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-workload-certs\") pod \"944451b3-0e62-4419-bf3a-37401268d08f\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") "
Oct 31 23:22:14.848644 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.836269    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istio-token\" (UniqueName: \"kubernetes.io/projected/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istio-token\") pod \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") "
Oct 31 23:22:14.848644 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.846088    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 23:22:14.888408 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.836349    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-istio-envoy\") pod \"944451b3-0e62-4419-bf3a-37401268d08f\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") "
Oct 31 23:22:14.888408 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.853593    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-workload-socket\") pod \"944451b3-0e62-4419-bf3a-37401268d08f\" (UID: \"944451b3-0e62-4419-bf3a-37401268d08f\") "
Oct 31 23:22:14.888408 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.853683    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istio-envoy\") pod \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\" (UID: \"de434cd6-3610-47ac-ae7e-78e1d05cb0c5\") "
Oct 31 23:22:14.888408 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.866191    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 23:22:14.888408 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.866887    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": context deadline exceeded"
Oct 31 23:22:14.888408 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.866988    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 23:22:14.888408 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.871483    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-credential-socket" pod "944451b3-0e62-4419-bf3a-37401268d08f" (UID: "944451b3-0e62-4419-bf3a-37401268d08f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:14.888408 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.875453    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-workload-socket" pod "de434cd6-3610-47ac-ae7e-78e1d05cb0c5" (UID: "de434cd6-3610-47ac-ae7e-78e1d05cb0c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:14.888408 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.881585    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/944451b3-0e62-4419-bf3a-37401268d08f-istio-podinfo" pod "944451b3-0e62-4419-bf3a-37401268d08f" (UID: "944451b3-0e62-4419-bf3a-37401268d08f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:22:14.888408 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.882200    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-workload-certs" pod "944451b3-0e62-4419-bf3a-37401268d08f" (UID: "944451b3-0e62-4419-bf3a-37401268d08f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:14.888408 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.888327    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-kube-api-access-kzfqk" pod "de434cd6-3610-47ac-ae7e-78e1d05cb0c5" (UID: "de434cd6-3610-47ac-ae7e-78e1d05cb0c5"). InnerVolumeSpecName "kube-api-access-kzfqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:22:14.889139 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.888502    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/944451b3-0e62-4419-bf3a-37401268d08f-istio-token" pod "944451b3-0e62-4419-bf3a-37401268d08f" (UID: "944451b3-0e62-4419-bf3a-37401268d08f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:22:14.931496 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.889634    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f421d513-57d6-43b1-af00-bed486a4d0bf-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:14.931496 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.889682    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:14.931496 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.889706    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/944451b3-0e62-4419-bf3a-37401268d08f-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:14.931496 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.889727    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:14.931496 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.889742    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzfqk\" (UniqueName: \"kubernetes.io/projected/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-kube-api-access-kzfqk\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:14.931496 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.889763    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/944451b3-0e62-4419-bf3a-37401268d08f-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:14.931496 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.889778    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:14.931496 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.889793    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:14.931496 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.889812    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:14.931496 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.889829    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:14.931496 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.889855    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/944451b3-0e62-4419-bf3a-37401268d08f-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:14.931496 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.889870    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:14.931496 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.889895    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:14.931496 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.889909    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:14.931496 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.898616    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istiod-ca-cert" pod "de434cd6-3610-47ac-ae7e-78e1d05cb0c5" (UID: "de434cd6-3610-47ac-ae7e-78e1d05cb0c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:22:14.931496 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:14.926551    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-workload-certs" pod "de434cd6-3610-47ac-ae7e-78e1d05cb0c5" (UID: "de434cd6-3610-47ac-ae7e-78e1d05cb0c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:15.025087 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.024963    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:15.118385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.081508    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istio-podinfo" pod "de434cd6-3610-47ac-ae7e-78e1d05cb0c5" (UID: "de434cd6-3610-47ac-ae7e-78e1d05cb0c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:22:15.169384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.142481    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-workload-socket" pod "944451b3-0e62-4419-bf3a-37401268d08f" (UID: "944451b3-0e62-4419-bf3a-37401268d08f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:15.260406 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.254170    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/944451b3-0e62-4419-bf3a-37401268d08f-kube-api-access-2dfl9" pod "944451b3-0e62-4419-bf3a-37401268d08f" (UID: "944451b3-0e62-4419-bf3a-37401268d08f"). InnerVolumeSpecName "kube-api-access-2dfl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:22:15.260406 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.255013    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-istio-envoy" pod "944451b3-0e62-4419-bf3a-37401268d08f" (UID: "944451b3-0e62-4419-bf3a-37401268d08f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:15.260406 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.255153    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:15.260406 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.256785    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f421d513-57d6-43b1-af00-bed486a4d0bf-istio-podinfo" pod "f421d513-57d6-43b1-af00-bed486a4d0bf" (UID: "f421d513-57d6-43b1-af00-bed486a4d0bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:22:15.260406 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.259575    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istio-token" pod "de434cd6-3610-47ac-ae7e-78e1d05cb0c5" (UID: "de434cd6-3610-47ac-ae7e-78e1d05cb0c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:22:15.261387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.261192    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:15.266546 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.262747    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dfl9\" (UniqueName: \"kubernetes.io/projected/944451b3-0e62-4419-bf3a-37401268d08f-kube-api-access-2dfl9\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:15.266546 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.262776    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:15.266546 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.262801    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:15.266546 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.262836    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f421d513-57d6-43b1-af00-bed486a4d0bf-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:15.266546 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.262855    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:15.266546 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.262870    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:15.266546 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.262885    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:15.266546 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.262907    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/944451b3-0e62-4419-bf3a-37401268d08f-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:15.276590 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.276501    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-istio-envoy" pod "f421d513-57d6-43b1-af00-bed486a4d0bf" (UID: "f421d513-57d6-43b1-af00-bed486a4d0bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:15.373028 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.372979    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97" podUID="c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:15.381781 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.381740    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f421d513-57d6-43b1-af00-bed486a4d0bf-istio-token" pod "f421d513-57d6-43b1-af00-bed486a4d0bf" (UID: "f421d513-57d6-43b1-af00-bed486a4d0bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:22:15.404608 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.404572    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istio-envoy" pod "de434cd6-3610-47ac-ae7e-78e1d05cb0c5" (UID: "de434cd6-3610-47ac-ae7e-78e1d05cb0c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:15.415349 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.415309    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de434cd6-3610-47ac-ae7e-78e1d05cb0c5-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:15.415349 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.415349    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f421d513-57d6-43b1-af00-bed486a4d0bf-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:15.437849 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.437817    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f421d513-57d6-43b1-af00-bed486a4d0bf-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:15.446119 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.445992    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f421d513-57d6-43b1-af00-bed486a4d0bf-kube-api-access-wscr6" pod "f421d513-57d6-43b1-af00-bed486a4d0bf" (UID: "f421d513-57d6-43b1-af00-bed486a4d0bf"). InnerVolumeSpecName "kube-api-access-wscr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:22:15.537819 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.537656    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 31 23:22:15.558731 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.539232    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"
Oct 31 23:22:15.641435 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.611617    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"
Oct 31 23:22:15.650834 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.650786    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wscr6\" (UniqueName: \"kubernetes.io/projected/f421d513-57d6-43b1-af00-bed486a4d0bf-kube-api-access-wscr6\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:15.713225 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.672023    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:15.780415 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.766736    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:22:15.823630 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.823573    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 31 23:22:15.899871 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.897677    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 23:22:15.899871 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.897992    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 23:22:15.920627 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.920562    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 23:22:15.981764 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.981728    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"
Oct 31 23:22:15.996267 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:15.996235    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:22:16.064152 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:16.064122    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:22:16.064574 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:16.064547    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94ecade3-w8zp7"]
Oct 31 23:22:16.067684 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:16.067659    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:22:16.067847 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:16.067818    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2" event={"ID":"f421d513-57d6-43b1-af00-bed486a4d0bf","Type":"ContainerDied","Data":"c66e6e07094b11a4e0cafe82215a981a290427f1d7e057e15fe4a9850c380f30"}
Oct 31 23:22:16.067965 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:16.067952    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm" event={"ID":"de434cd6-3610-47ac-ae7e-78e1d05cb0c5","Type":"ContainerDied","Data":"38d85dce31df8e57567143370194c087cc263b08e73ccb840c072ee7a6ab70c2"}
Oct 31 23:22:16.068056 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:16.068043    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs" event={"ID":"944451b3-0e62-4419-bf3a-37401268d08f","Type":"ContainerDied","Data":"9fa108f7d512f5d0f0a126a321daa4903a0c48314c76c623104f9ca066e9cd3d"}
Oct 31 23:22:16.073438 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:16.068163    4301 scope.go:122] "RemoveContainer" containerID="26c66833d17c4de159bed0375522a142211248db1eb912d9c40bbce7f7f97004"
Oct 31 23:22:16.073803 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:16.025540    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded"
Oct 31 23:22:16.074125 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:16.030193    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 31 23:22:16.106171 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:22:16.106030    4301 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf421d513_57d6_43b1_af00_bed486a4d0bf.slice/crio-c66e6e07094b11a4e0cafe82215a981a290427f1d7e057e15fe4a9850c380f30\": RecentStats: unable to find data in memory cache]"
Oct 31 23:22:16.423820 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:16.417040    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97" podUID="c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:16.423820 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:16.394875    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:16.423820 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:16.332685    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 23:22:16.462737 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:16.462568    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"
Oct 31 23:22:16.763664 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:16.733823    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:16.807379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:16.800439    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:16.828827 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:16.828220    4301 scope.go:122] "RemoveContainer" containerID="8342fb6cfa9189bbefe86a43bbe1878ce7caae656611eeb74aac34a0ee189a9c"
Oct 31 23:22:16.829023 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:16.828920    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded"
Oct 31 23:22:16.918413 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:16.915946    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 23:22:16.918413 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:16.917262    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:16.918413 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:16.917636    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 23:22:16.934475 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:16.934432    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 23:22:16.990586 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:16.990487    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"]
Oct 31 23:22:17.315249 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:17.080858    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:17.315249 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:17.082860    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-lhcv2"]
Oct 31 23:22:17.315249 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:17.260070    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ecade3-w8zp7" event={"ID":"05c2fd23-331c-4818-b14c-786d6e14477a","Type":"ContainerStarted","Data":"2e2320bf9c2d429031b5e942a2cbce2911b1825d80eecb379fd98cfa7e497645"}
Oct 31 23:22:17.315249 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:17.301100    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:17.342490 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:17.342403    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:17.449386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:17.438718    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 31 23:22:17.487379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:17.449880    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 23:22:17.656484 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:17.617236    4301 scope.go:122] "RemoveContainer" containerID="fb09bceafe3fe9ee13ce54940d702896cd7be3b03819ead1f800208d4b19f688"
Oct 31 23:22:17.660238 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:17.648554    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:17.755970 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:17.755927    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:17.824424 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:17.810688    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 23:22:17.824424 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:17.811251    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97" podUID="c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 31 23:22:17.902956 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:17.902905    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f421d513-57d6-43b1-af00-bed486a4d0bf" path="/var/lib/kubelet/pods/f421d513-57d6-43b1-af00-bed486a4d0bf/volumes"
Oct 31 23:22:17.903795 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:17.903775    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"]
Oct 31 23:22:17.936387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:17.930714    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:17.936387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:17.931096    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:17.936387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:17.931336    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 31 23:22:17.999492 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:17.999456    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-gsdxm"]
Oct 31 23:22:18.009936 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:18.006718    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:18.033406 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:18.019009    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:18.098093 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:18.089676    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:22:18.153949 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:18.123498    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:18.181872 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:18.180147    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:22:18.420325 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:18.420288    4301 generic.go:356] "Generic (PLEG): container finished" podID="05c2fd23-331c-4818-b14c-786d6e14477a" containerID="252166cfcd8cbb9a1615a6065a46d543422b1bad7ee2a56d7fd60ba7b592d104" exitCode=0
Oct 31 23:22:18.420726 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:18.420698    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ecade3-w8zp7" event={"ID":"05c2fd23-331c-4818-b14c-786d6e14477a","Type":"ContainerDied","Data":"252166cfcd8cbb9a1615a6065a46d543422b1bad7ee2a56d7fd60ba7b592d104"}
Oct 31 23:22:18.482528 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:18.482465    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 31 23:22:18.501061 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:18.501017    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:18.634950 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:18.634918    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:22:18.650957 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:18.650775    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:22:18.655589 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:18.655566    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:22:18.655965 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:18.655951    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:22:18.656114 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:18.656104    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:22:18.657560 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:18.657498    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:22:18.672165 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:18.672120    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"]
Oct 31 23:22:18.700946 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:18.700911    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-b2tqs"]
Oct 31 23:22:18.733525 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:18.733495    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:22:19.492109 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:19.491763    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="944451b3-0e62-4419-bf3a-37401268d08f" path="/var/lib/kubelet/pods/944451b3-0e62-4419-bf3a-37401268d08f/volumes"
Oct 31 23:22:19.492809 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:19.492768    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de434cd6-3610-47ac-ae7e-78e1d05cb0c5" path="/var/lib/kubelet/pods/de434cd6-3610-47ac-ae7e-78e1d05cb0c5/volumes"
Oct 31 23:22:19.864256 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:19.864222    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ecade3-w8zp7"
Oct 31 23:22:19.935848 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:19.935772    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05c2fd23-331c-4818-b14c-786d6e14477a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05c2fd23-331c-4818-b14c-786d6e14477a-maas-api-openshift-ca\") pod \"05c2fd23-331c-4818-b14c-786d6e14477a\" (UID: \"05c2fd23-331c-4818-b14c-786d6e14477a\") "
Oct 31 23:22:19.935848 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:19.935799    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05c2fd23-331c-4818-b14c-786d6e14477a-maas-api-openshift-ca" pod "05c2fd23-331c-4818-b14c-786d6e14477a" (UID: "05c2fd23-331c-4818-b14c-786d6e14477a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:22:19.936098 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:19.935885    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05c2fd23-331c-4818-b14c-786d6e14477a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05c2fd23-331c-4818-b14c-786d6e14477a-maas-api-xks-ca\") pod \"05c2fd23-331c-4818-b14c-786d6e14477a\" (UID: \"05c2fd23-331c-4818-b14c-786d6e14477a\") "
Oct 31 23:22:19.936098 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:19.936034    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05c2fd23-331c-4818-b14c-786d6e14477a-kube-api-access-65mc2\" (UniqueName: \"kubernetes.io/projected/05c2fd23-331c-4818-b14c-786d6e14477a-kube-api-access-65mc2\") pod \"05c2fd23-331c-4818-b14c-786d6e14477a\" (UID: \"05c2fd23-331c-4818-b14c-786d6e14477a\") "
Oct 31 23:22:19.937569 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:19.937543    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05c2fd23-331c-4818-b14c-786d6e14477a-maas-api-openshift-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:19.939969 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:19.939942    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05c2fd23-331c-4818-b14c-786d6e14477a-maas-api-xks-ca" pod "05c2fd23-331c-4818-b14c-786d6e14477a" (UID: "05c2fd23-331c-4818-b14c-786d6e14477a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:22:19.944223 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:19.944194    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c2fd23-331c-4818-b14c-786d6e14477a-kube-api-access-65mc2" pod "05c2fd23-331c-4818-b14c-786d6e14477a" (UID: "05c2fd23-331c-4818-b14c-786d6e14477a"). InnerVolumeSpecName "kube-api-access-65mc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:22:20.039775 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:20.039728    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05c2fd23-331c-4818-b14c-786d6e14477a-maas-api-xks-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:20.039775 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:20.039774    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65mc2\" (UniqueName: \"kubernetes.io/projected/05c2fd23-331c-4818-b14c-786d6e14477a-kube-api-access-65mc2\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:20.520737 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:20.520668    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ecade3-w8zp7" event={"ID":"05c2fd23-331c-4818-b14c-786d6e14477a","Type":"ContainerDied","Data":"2e2320bf9c2d429031b5e942a2cbce2911b1825d80eecb379fd98cfa7e497645"}
Oct 31 23:22:20.521139 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:20.520746    4301 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e2320bf9c2d429031b5e942a2cbce2911b1825d80eecb379fd98cfa7e497645"
Oct 31 23:22:20.521139 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:20.520828    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ecade3-w8zp7"
Oct 31 23:22:20.847157 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:20.846669    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e595be3-jrjfx"]
Oct 31 23:22:20.848795 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:20.848561    4301 cpu_manager.go:414] "containerMap: removing container" podUID="f421d513-57d6-43b1-af00-bed486a4d0bf" containerName="istio-proxy"
Oct 31 23:22:20.848795 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:20.848589    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f421d513-57d6-43b1-af00-bed486a4d0bf" containerName="istio-proxy"
Oct 31 23:22:20.848795 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:20.848617    4301 cpu_manager.go:414] "containerMap: removing container" podUID="05c2fd23-331c-4818-b14c-786d6e14477a" containerName="revoke-keys"
Oct 31 23:22:20.848795 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:20.848625    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05c2fd23-331c-4818-b14c-786d6e14477a" containerName="revoke-keys"
Oct 31 23:22:20.848795 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:20.848660    4301 cpu_manager.go:414] "containerMap: removing container" podUID="de434cd6-3610-47ac-ae7e-78e1d05cb0c5" containerName="istio-proxy"
Oct 31 23:22:20.848795 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:20.848668    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de434cd6-3610-47ac-ae7e-78e1d05cb0c5" containerName="istio-proxy"
Oct 31 23:22:20.852619 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:20.849272    4301 cpu_manager.go:414] "containerMap: removing container" podUID="944451b3-0e62-4419-bf3a-37401268d08f" containerName="istio-proxy"
Oct 31 23:22:20.852619 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:20.849292    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="944451b3-0e62-4419-bf3a-37401268d08f" containerName="istio-proxy"
Oct 31 23:22:20.852619 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:20.849674    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="944451b3-0e62-4419-bf3a-37401268d08f" containerName="istio-proxy"
Oct 31 23:22:20.852619 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:20.849706    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="f421d513-57d6-43b1-af00-bed486a4d0bf" containerName="istio-proxy"
Oct 31 23:22:20.852619 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:20.849726    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="05c2fd23-331c-4818-b14c-786d6e14477a" containerName="revoke-keys"
Oct 31 23:22:20.852619 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:20.849746    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="de434cd6-3610-47ac-ae7e-78e1d05cb0c5" containerName="istio-proxy"
Oct 31 23:22:20.856464 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:20.855114    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e595be3-jrjfx"
Oct 31 23:22:20.861108 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:20.861080    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e595be3-jrjfx"]
Oct 31 23:22:20.868270 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:20.868248    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\""
Oct 31 23:22:20.868570 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:20.868554    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 23:22:20.976115 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:20.974943    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tctb\" (UniqueName: \"kubernetes.io/projected/53842309-02b2-4ce4-a9e6-334334c57b84-kube-api-access-8tctb\") pod \"maas-api-revoke-sub-5e595be3-jrjfx\" (UID: \"53842309-02b2-4ce4-a9e6-334334c57b84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e595be3-jrjfx"
Oct 31 23:22:20.976115 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:20.975002    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53842309-02b2-4ce4-a9e6-334334c57b84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e595be3-jrjfx\" (UID: \"53842309-02b2-4ce4-a9e6-334334c57b84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e595be3-jrjfx"
Oct 31 23:22:20.976115 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:20.975108    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53842309-02b2-4ce4-a9e6-334334c57b84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e595be3-jrjfx\" (UID: \"53842309-02b2-4ce4-a9e6-334334c57b84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e595be3-jrjfx"
Oct 31 23:22:21.099428 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:21.090885    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tctb\" (UniqueName: \"kubernetes.io/projected/53842309-02b2-4ce4-a9e6-334334c57b84-kube-api-access-8tctb\") pod \"maas-api-revoke-sub-5e595be3-jrjfx\" (UID: \"53842309-02b2-4ce4-a9e6-334334c57b84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e595be3-jrjfx"
Oct 31 23:22:21.099428 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:21.090941    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53842309-02b2-4ce4-a9e6-334334c57b84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e595be3-jrjfx\" (UID: \"53842309-02b2-4ce4-a9e6-334334c57b84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e595be3-jrjfx"
Oct 31 23:22:21.099428 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:21.091063    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53842309-02b2-4ce4-a9e6-334334c57b84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e595be3-jrjfx\" (UID: \"53842309-02b2-4ce4-a9e6-334334c57b84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e595be3-jrjfx"
Oct 31 23:22:21.110604 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:21.101004    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53842309-02b2-4ce4-a9e6-334334c57b84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e595be3-jrjfx\" (UID: \"53842309-02b2-4ce4-a9e6-334334c57b84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e595be3-jrjfx"
Oct 31 23:22:21.133516 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:21.128597    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tctb\" (UniqueName: \"kubernetes.io/projected/53842309-02b2-4ce4-a9e6-334334c57b84-kube-api-access-8tctb\") pod \"maas-api-revoke-sub-5e595be3-jrjfx\" (UID: \"53842309-02b2-4ce4-a9e6-334334c57b84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e595be3-jrjfx"
Oct 31 23:22:21.145769 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:21.140899    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53842309-02b2-4ce4-a9e6-334334c57b84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e595be3-jrjfx\" (UID: \"53842309-02b2-4ce4-a9e6-334334c57b84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e595be3-jrjfx"
Oct 31 23:22:21.196609 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:21.196305    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e595be3-jrjfx"
Oct 31 23:22:21.432594 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:21.432551    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4528ed-4hkf5"]
Oct 31 23:22:21.490146 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:21.490079    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4528ed-4hkf5"]
Oct 31 23:22:21.490328 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:21.490179    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4528ed-4hkf5"
Oct 31 23:22:21.600632 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:21.600583    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/893091bf-0eb6-4443-8060-b35d05383b05-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4528ed-4hkf5\" (UID: \"893091bf-0eb6-4443-8060-b35d05383b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4528ed-4hkf5"
Oct 31 23:22:21.601158 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:21.600751    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkkc6\" (UniqueName: \"kubernetes.io/projected/893091bf-0eb6-4443-8060-b35d05383b05-kube-api-access-jkkc6\") pod \"maas-api-revoke-keys-e2e-api-a-4528ed-4hkf5\" (UID: \"893091bf-0eb6-4443-8060-b35d05383b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4528ed-4hkf5"
Oct 31 23:22:21.601245 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:21.601172    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/893091bf-0eb6-4443-8060-b35d05383b05-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4528ed-4hkf5\" (UID: \"893091bf-0eb6-4443-8060-b35d05383b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4528ed-4hkf5"
Oct 31 23:22:21.625097 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:21.625064    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e595be3-jrjfx"]
Oct 31 23:22:21.625232 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:22:21.625073    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53842309_02b2_4ce4_a9e6_334334c57b84.slice/crio-111cc9eca3cc5b3990fbe5d55177e1f85cb225635a47255506f207bc869d0f0b WatchSource:0}: Error finding container 111cc9eca3cc5b3990fbe5d55177e1f85cb225635a47255506f207bc869d0f0b: Status 404 returned error can't find the container with id 111cc9eca3cc5b3990fbe5d55177e1f85cb225635a47255506f207bc869d0f0b
Oct 31 23:22:21.642336 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:21.642094    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-479107-6d88b6b77d-hzkkg"]
Oct 31 23:22:21.654494 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:21.651261    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-479107-748df6cc67-5hrhg"]
Oct 31 23:22:21.703223 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:21.703187    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/893091bf-0eb6-4443-8060-b35d05383b05-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4528ed-4hkf5\" (UID: \"893091bf-0eb6-4443-8060-b35d05383b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4528ed-4hkf5"
Oct 31 23:22:21.703435 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:21.703287    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkkc6\" (UniqueName: \"kubernetes.io/projected/893091bf-0eb6-4443-8060-b35d05383b05-kube-api-access-jkkc6\") pod \"maas-api-revoke-keys-e2e-api-a-4528ed-4hkf5\" (UID: \"893091bf-0eb6-4443-8060-b35d05383b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4528ed-4hkf5"
Oct 31 23:22:21.703518 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:21.703473    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/893091bf-0eb6-4443-8060-b35d05383b05-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4528ed-4hkf5\" (UID: \"893091bf-0eb6-4443-8060-b35d05383b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4528ed-4hkf5"
Oct 31 23:22:21.704331 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:21.704293    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/893091bf-0eb6-4443-8060-b35d05383b05-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4528ed-4hkf5\" (UID: \"893091bf-0eb6-4443-8060-b35d05383b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4528ed-4hkf5"
Oct 31 23:22:21.730798 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:21.726760    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/893091bf-0eb6-4443-8060-b35d05383b05-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4528ed-4hkf5\" (UID: \"893091bf-0eb6-4443-8060-b35d05383b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4528ed-4hkf5"
Oct 31 23:22:21.734241 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:21.734205    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkkc6\" (UniqueName: \"kubernetes.io/projected/893091bf-0eb6-4443-8060-b35d05383b05-kube-api-access-jkkc6\") pod \"maas-api-revoke-keys-e2e-api-a-4528ed-4hkf5\" (UID: \"893091bf-0eb6-4443-8060-b35d05383b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4528ed-4hkf5"
Oct 31 23:22:21.841544 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:21.841500    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4528ed-4hkf5"
Oct 31 23:22:22.600426 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:22.566772    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e595be3-jrjfx" event={"ID":"53842309-02b2-4ce4-a9e6-334334c57b84","Type":"ContainerStarted","Data":"2cd5f93792ccafcd8dc80c5a888176640cacecd5b4afb878b9c466d28662e0da"}
Oct 31 23:22:22.600426 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:22.566881    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e595be3-jrjfx" event={"ID":"53842309-02b2-4ce4-a9e6-334334c57b84","Type":"ContainerStarted","Data":"111cc9eca3cc5b3990fbe5d55177e1f85cb225635a47255506f207bc869d0f0b"}
Oct 31 23:22:22.612285 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:22.612180    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e595be3-jrjfx" podStartSLOduration=2.612159601 podStartE2EDuration="2.612159601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:22:22.606624313 +0000 UTC m=+1961.353813818" watchObservedRunningTime="2026-10-31 23:22:22.612159601 +0000 UTC m=+1961.359349107"
Oct 31 23:22:22.645401 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:22.645336    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4528ed-4hkf5"]
Oct 31 23:22:23.600001 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:23.599901    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4528ed-4hkf5" event={"ID":"893091bf-0eb6-4443-8060-b35d05383b05","Type":"ContainerStarted","Data":"72fe69afc94e8b31ecbc159e8d5d7bb18793b87acd3fb4583c46aa95172cb073"}
Oct 31 23:22:23.600001 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:23.599977    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4528ed-4hkf5" event={"ID":"893091bf-0eb6-4443-8060-b35d05383b05","Type":"ContainerStarted","Data":"91275d84502fb40665928634ce01cfa27a38aea20c8956a10811a9665b85527c"}
Oct 31 23:22:23.603910 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:23.603875    4301 generic.go:356] "Generic (PLEG): container finished" podID="53842309-02b2-4ce4-a9e6-334334c57b84" containerID="2cd5f93792ccafcd8dc80c5a888176640cacecd5b4afb878b9c466d28662e0da" exitCode=0
Oct 31 23:22:23.604061 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:23.603965    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e595be3-jrjfx" event={"ID":"53842309-02b2-4ce4-a9e6-334334c57b84","Type":"ContainerDied","Data":"2cd5f93792ccafcd8dc80c5a888176640cacecd5b4afb878b9c466d28662e0da"}
Oct 31 23:22:23.668427 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:23.665927    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4528ed-4hkf5" podStartSLOduration=2.665904361 podStartE2EDuration="2.665904361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:22:23.62287826 +0000 UTC m=+1962.370067768" watchObservedRunningTime="2026-10-31 23:22:23.665904361 +0000 UTC m=+1962.413093866"
Oct 31 23:22:23.803551 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:23.803505    4301 patch_prober.go:29] interesting pod/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 23:22:23.803785 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:23.803571    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62" podUID="d05b75f2-18af-4d86-99d9-a51d2d23c3b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:24.033164 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:24.032825    4301 patch_prober.go:29] interesting pod/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 23:22:24.033164 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:24.032892    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x" podUID="74a97819-25d1-43c8-b104-ee21999c5f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:24.716146 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:24.715424    4301 generic.go:356] "Generic (PLEG): container finished" podID="893091bf-0eb6-4443-8060-b35d05383b05" containerID="72fe69afc94e8b31ecbc159e8d5d7bb18793b87acd3fb4583c46aa95172cb073" exitCode=0
Oct 31 23:22:24.716146 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:24.716058    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4528ed-4hkf5" event={"ID":"893091bf-0eb6-4443-8060-b35d05383b05","Type":"ContainerDied","Data":"72fe69afc94e8b31ecbc159e8d5d7bb18793b87acd3fb4583c46aa95172cb073"}
Oct 31 23:22:25.107181 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:25.106763    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e595be3-jrjfx"
Oct 31 23:22:25.338849 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:25.239108    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53842309-02b2-4ce4-a9e6-334334c57b84-kube-api-access-8tctb\" (UniqueName: \"kubernetes.io/projected/53842309-02b2-4ce4-a9e6-334334c57b84-kube-api-access-8tctb\") pod \"53842309-02b2-4ce4-a9e6-334334c57b84\" (UID: \"53842309-02b2-4ce4-a9e6-334334c57b84\") "
Oct 31 23:22:25.338849 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:25.239274    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53842309-02b2-4ce4-a9e6-334334c57b84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53842309-02b2-4ce4-a9e6-334334c57b84-maas-api-openshift-ca\") pod \"53842309-02b2-4ce4-a9e6-334334c57b84\" (UID: \"53842309-02b2-4ce4-a9e6-334334c57b84\") "
Oct 31 23:22:25.338849 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:25.239319    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53842309-02b2-4ce4-a9e6-334334c57b84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53842309-02b2-4ce4-a9e6-334334c57b84-maas-api-xks-ca\") pod \"53842309-02b2-4ce4-a9e6-334334c57b84\" (UID: \"53842309-02b2-4ce4-a9e6-334334c57b84\") "
Oct 31 23:22:25.347627 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:25.347261    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53842309-02b2-4ce4-a9e6-334334c57b84-maas-api-openshift-ca" pod "53842309-02b2-4ce4-a9e6-334334c57b84" (UID: "53842309-02b2-4ce4-a9e6-334334c57b84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:22:25.350460 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:25.350431    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53842309-02b2-4ce4-a9e6-334334c57b84-maas-api-openshift-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:25.395762 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:25.395721    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53842309-02b2-4ce4-a9e6-334334c57b84-maas-api-xks-ca" pod "53842309-02b2-4ce4-a9e6-334334c57b84" (UID: "53842309-02b2-4ce4-a9e6-334334c57b84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:22:25.395936 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:25.395854    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53842309-02b2-4ce4-a9e6-334334c57b84-kube-api-access-8tctb" pod "53842309-02b2-4ce4-a9e6-334334c57b84" (UID: "53842309-02b2-4ce4-a9e6-334334c57b84"). InnerVolumeSpecName "kube-api-access-8tctb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:22:25.478186 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:25.478145    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tctb\" (UniqueName: \"kubernetes.io/projected/53842309-02b2-4ce4-a9e6-334334c57b84-kube-api-access-8tctb\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:25.478186 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:25.478194    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53842309-02b2-4ce4-a9e6-334334c57b84-maas-api-xks-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:26.185045 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:26.178109    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e595be3-jrjfx"
Oct 31 23:22:26.185045 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:26.178926    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e595be3-jrjfx" event={"ID":"53842309-02b2-4ce4-a9e6-334334c57b84","Type":"ContainerDied","Data":"111cc9eca3cc5b3990fbe5d55177e1f85cb225635a47255506f207bc869d0f0b"}
Oct 31 23:22:26.185045 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:26.178996    4301 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="111cc9eca3cc5b3990fbe5d55177e1f85cb225635a47255506f207bc869d0f0b"
Oct 31 23:22:26.663402 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:26.651422    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-479107-6d88b6b77d-hzkkg" podUID="cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27" containerName="payload-processing" containerID="cri-o://cc0d97e3dcacb25b42b3d6d5870bef3b196bdbe80c6f38266d7919e08bfc2b13" gracePeriod=25
Oct 31 23:22:26.663402 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:26.652793    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-479107-748df6cc67-5hrhg" podUID="bb1cddef-0805-49a6-86dc-4489565710bd" containerName="payload-pre-processing" containerID="cri-o://8c87951594dd210ed9fb07437565df5387aca7630f3615a4afdde98d9ee2e26d" gracePeriod=25
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:26.959578    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4528ed-4hkf5"
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.065305    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/893091bf-0eb6-4443-8060-b35d05383b05-kube-api-access-jkkc6\" (UniqueName: \"kubernetes.io/projected/893091bf-0eb6-4443-8060-b35d05383b05-kube-api-access-jkkc6\") pod \"893091bf-0eb6-4443-8060-b35d05383b05\" (UID: \"893091bf-0eb6-4443-8060-b35d05383b05\") "
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.065994    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/893091bf-0eb6-4443-8060-b35d05383b05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/893091bf-0eb6-4443-8060-b35d05383b05-maas-api-openshift-ca\") pod \"893091bf-0eb6-4443-8060-b35d05383b05\" (UID: \"893091bf-0eb6-4443-8060-b35d05383b05\") "
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.066431    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/893091bf-0eb6-4443-8060-b35d05383b05-maas-api-openshift-ca" pod "893091bf-0eb6-4443-8060-b35d05383b05" (UID: "893091bf-0eb6-4443-8060-b35d05383b05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.066825    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/893091bf-0eb6-4443-8060-b35d05383b05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/893091bf-0eb6-4443-8060-b35d05383b05-maas-api-xks-ca\") pod \"893091bf-0eb6-4443-8060-b35d05383b05\" (UID: \"893091bf-0eb6-4443-8060-b35d05383b05\") "
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.067805    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/893091bf-0eb6-4443-8060-b35d05383b05-maas-api-openshift-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.069512    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/893091bf-0eb6-4443-8060-b35d05383b05-kube-api-access-jkkc6" pod "893091bf-0eb6-4443-8060-b35d05383b05" (UID: "893091bf-0eb6-4443-8060-b35d05383b05"). InnerVolumeSpecName "kube-api-access-jkkc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.071999    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/893091bf-0eb6-4443-8060-b35d05383b05-maas-api-xks-ca" pod "893091bf-0eb6-4443-8060-b35d05383b05" (UID: "893091bf-0eb6-4443-8060-b35d05383b05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.114103    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-479107-6d88b6b77d-hzkkg"
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.170058    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/893091bf-0eb6-4443-8060-b35d05383b05-maas-api-xks-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.170086    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkkc6\" (UniqueName: \"kubernetes.io/projected/893091bf-0eb6-4443-8060-b35d05383b05-kube-api-access-jkkc6\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.194792    4301 generic.go:356] "Generic (PLEG): container finished" podID="cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27" containerID="cc0d97e3dcacb25b42b3d6d5870bef3b196bdbe80c6f38266d7919e08bfc2b13" exitCode=0
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.194838    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-479107-6d88b6b77d-hzkkg"
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.195001    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-479107-6d88b6b77d-hzkkg" event={"ID":"cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27","Type":"ContainerDied","Data":"cc0d97e3dcacb25b42b3d6d5870bef3b196bdbe80c6f38266d7919e08bfc2b13"}
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.195053    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-479107-6d88b6b77d-hzkkg" event={"ID":"cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27","Type":"ContainerDied","Data":"ea1bb8a6d794a6931d0b17d6f279facc19bff7c9d87ed5b052d570a7458e29e5"}
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.195084    4301 scope.go:122] "RemoveContainer" containerID="cc0d97e3dcacb25b42b3d6d5870bef3b196bdbe80c6f38266d7919e08bfc2b13"
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.197715    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4528ed-4hkf5" event={"ID":"893091bf-0eb6-4443-8060-b35d05383b05","Type":"ContainerDied","Data":"91275d84502fb40665928634ce01cfa27a38aea20c8956a10811a9665b85527c"}
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.197759    4301 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91275d84502fb40665928634ce01cfa27a38aea20c8956a10811a9665b85527c"
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.197814    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4528ed-4hkf5"
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.207431    4301 generic.go:356] "Generic (PLEG): container finished" podID="bb1cddef-0805-49a6-86dc-4489565710bd" containerID="8c87951594dd210ed9fb07437565df5387aca7630f3615a4afdde98d9ee2e26d" exitCode=0
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.207509    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-479107-748df6cc67-5hrhg" event={"ID":"bb1cddef-0805-49a6-86dc-4489565710bd","Type":"ContainerDied","Data":"8c87951594dd210ed9fb07437565df5387aca7630f3615a4afdde98d9ee2e26d"}
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.210695    4301 generic.go:356] "Generic (PLEG): container finished" podID="46881476-3103-4ee9-9b90-ec0ba9f30c3e" containerID="876cb199f3051d7a7d6abea4aba1b81d36582ea4122d5e6e269e4ca44106574d" exitCode=0
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.210786    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7" event={"ID":"46881476-3103-4ee9-9b90-ec0ba9f30c3e","Type":"ContainerDied","Data":"876cb199f3051d7a7d6abea4aba1b81d36582ea4122d5e6e269e4ca44106574d"}
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.211211    4301 scope.go:122] "RemoveContainer" containerID="876cb199f3051d7a7d6abea4aba1b81d36582ea4122d5e6e269e4ca44106574d"
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.211758    4301 scope.go:122] "RemoveContainer" containerID="cc0d97e3dcacb25b42b3d6d5870bef3b196bdbe80c6f38266d7919e08bfc2b13"
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:22:27.212338    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc0d97e3dcacb25b42b3d6d5870bef3b196bdbe80c6f38266d7919e08bfc2b13\": container with ID starting with cc0d97e3dcacb25b42b3d6d5870bef3b196bdbe80c6f38266d7919e08bfc2b13 not found: ID does not exist" containerID="cc0d97e3dcacb25b42b3d6d5870bef3b196bdbe80c6f38266d7919e08bfc2b13"
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.212424    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc0d97e3dcacb25b42b3d6d5870bef3b196bdbe80c6f38266d7919e08bfc2b13"} err="failed to get container status \"cc0d97e3dcacb25b42b3d6d5870bef3b196bdbe80c6f38266d7919e08bfc2b13\": rpc error: code = NotFound desc = could not find container \"cc0d97e3dcacb25b42b3d6d5870bef3b196bdbe80c6f38266d7919e08bfc2b13\": container with ID starting with cc0d97e3dcacb25b42b3d6d5870bef3b196bdbe80c6f38266d7919e08bfc2b13 not found: ID does not exist"
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.215296    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.215447    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.215513    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.226006    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-479107-748df6cc67-5hrhg"
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.272183    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27-plugins-config-volume\") pod \"cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27\" (UID: \"cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27\") "
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.272377    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27-kube-api-access-hfqgj\" (UniqueName: \"kubernetes.io/projected/cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27-kube-api-access-hfqgj\") pod \"cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27\" (UID: \"cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27\") "
Oct 31 23:22:27.280069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.275603    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27-plugins-config-volume" pod "cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27" (UID: "cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:22:27.288228 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.288187    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27-kube-api-access-hfqgj" pod "cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27" (UID: "cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27"). InnerVolumeSpecName "kube-api-access-hfqgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:22:27.395415 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.382301    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb1cddef-0805-49a6-86dc-4489565710bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb1cddef-0805-49a6-86dc-4489565710bd-plugins-config-volume\") pod \"bb1cddef-0805-49a6-86dc-4489565710bd\" (UID: \"bb1cddef-0805-49a6-86dc-4489565710bd\") "
Oct 31 23:22:27.395415 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.382460    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb1cddef-0805-49a6-86dc-4489565710bd-kube-api-access-lfdfw\" (UniqueName: \"kubernetes.io/projected/bb1cddef-0805-49a6-86dc-4489565710bd-kube-api-access-lfdfw\") pod \"bb1cddef-0805-49a6-86dc-4489565710bd\" (UID: \"bb1cddef-0805-49a6-86dc-4489565710bd\") "
Oct 31 23:22:27.395415 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.383470    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfqgj\" (UniqueName: \"kubernetes.io/projected/cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27-kube-api-access-hfqgj\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:27.395415 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.383491    4301 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27-plugins-config-volume\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:27.395415 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.394727    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb1cddef-0805-49a6-86dc-4489565710bd-plugins-config-volume" pod "bb1cddef-0805-49a6-86dc-4489565710bd" (UID: "bb1cddef-0805-49a6-86dc-4489565710bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:22:27.494689 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.494646    4301 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb1cddef-0805-49a6-86dc-4489565710bd-plugins-config-volume\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:27.598381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.595226    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb1cddef-0805-49a6-86dc-4489565710bd-kube-api-access-lfdfw" pod "bb1cddef-0805-49a6-86dc-4489565710bd" (UID: "bb1cddef-0805-49a6-86dc-4489565710bd"). InnerVolumeSpecName "kube-api-access-lfdfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:22:27.598381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.598101    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfdfw\" (UniqueName: \"kubernetes.io/projected/bb1cddef-0805-49a6-86dc-4489565710bd-kube-api-access-lfdfw\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:27.615510 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.614929    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-479107-6d88b6b77d-hzkkg"]
Oct 31 23:22:27.622342 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.622314    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-479107-6d88b6b77d-hzkkg"]
Oct 31 23:22:27.622716 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.622697    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": read tcp 10.133.0.2:50144->10.133.0.100:15021: read: connection reset by peer" start-of-body=
Oct 31 23:22:27.622757 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:27.622736    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p" podUID="394e1575-7171-45a2-8d19-ef551eaad0ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": read tcp 10.133.0.2:50144->10.133.0.100:15021: read: connection reset by peer"
Oct 31 23:22:28.235387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:28.225996    4301 generic.go:356] "Generic (PLEG): container finished" podID="74a97819-25d1-43c8-b104-ee21999c5f68" containerID="9457e388751789f92d5088d58f4cd2dda892459c3ddd36288972da7dea82957f" exitCode=0
Oct 31 23:22:28.235387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:28.226095    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x" event={"ID":"74a97819-25d1-43c8-b104-ee21999c5f68","Type":"ContainerDied","Data":"9457e388751789f92d5088d58f4cd2dda892459c3ddd36288972da7dea82957f"}
Oct 31 23:22:28.235387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:28.228138    4301 scope.go:122] "RemoveContainer" containerID="9457e388751789f92d5088d58f4cd2dda892459c3ddd36288972da7dea82957f"
Oct 31 23:22:28.235387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:28.228384    4301 generic.go:356] "Generic (PLEG): container finished" podID="394e1575-7171-45a2-8d19-ef551eaad0ea" containerID="0f85ac85e722e495f38549d8df0595653b60c874b0a2047a4ac9e9aed8bf6864" exitCode=0
Oct 31 23:22:28.235387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:28.228488    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p" event={"ID":"394e1575-7171-45a2-8d19-ef551eaad0ea","Type":"ContainerDied","Data":"0f85ac85e722e495f38549d8df0595653b60c874b0a2047a4ac9e9aed8bf6864"}
Oct 31 23:22:28.235387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:28.228962    4301 scope.go:122] "RemoveContainer" containerID="0f85ac85e722e495f38549d8df0595653b60c874b0a2047a4ac9e9aed8bf6864"
Oct 31 23:22:28.235387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:28.231482    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:22:28.235387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:28.231568    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:22:28.235387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:28.231614    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:22:28.235387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:28.231779    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:22:28.235387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:28.231858    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:22:28.235387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:28.231915    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:22:28.242647 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:28.242606    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-479107-748df6cc67-5hrhg" event={"ID":"bb1cddef-0805-49a6-86dc-4489565710bd","Type":"ContainerDied","Data":"5b52177371ee4f02c03ca38bed43de4c247420ecf71e9decaf12ea2c0955e371"}
Oct 31 23:22:28.242783 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:28.242680    4301 scope.go:122] "RemoveContainer" containerID="8c87951594dd210ed9fb07437565df5387aca7630f3615a4afdde98d9ee2e26d"
Oct 31 23:22:28.242843 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:28.242813    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-479107-748df6cc67-5hrhg"
Oct 31 23:22:28.248831 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:28.248799    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7" event={"ID":"46881476-3103-4ee9-9b90-ec0ba9f30c3e","Type":"ContainerStarted","Data":"f27c427c52e9908a305179ea8354e0690e33ac07fa37c31b64fe479465bd5033"}
Oct 31 23:22:28.309731 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:28.308221    4301 generic.go:356] "Generic (PLEG): container finished" podID="d05b75f2-18af-4d86-99d9-a51d2d23c3b5" containerID="4a30b386bc535202ee7f79b1b6b7bbceaac019d88ca04099ae4c0367bf107eba" exitCode=0
Oct 31 23:22:28.309731 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:28.308316    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62" event={"ID":"d05b75f2-18af-4d86-99d9-a51d2d23c3b5","Type":"ContainerDied","Data":"4a30b386bc535202ee7f79b1b6b7bbceaac019d88ca04099ae4c0367bf107eba"}
Oct 31 23:22:28.309731 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:28.308925    4301 scope.go:122] "RemoveContainer" containerID="4a30b386bc535202ee7f79b1b6b7bbceaac019d88ca04099ae4c0367bf107eba"
Oct 31 23:22:28.325049 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:28.325010    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:22:28.325207 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:28.325091    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:22:28.325207 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:28.325127    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:22:28.498037 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:28.497449    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-479107-748df6cc67-5hrhg"]
Oct 31 23:22:28.509570 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:28.509530    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-479107-748df6cc67-5hrhg"]
Oct 31 23:22:29.094447 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:29.093286    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:22:29.095187 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:29.094897    4301 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 31 23:22:29.095187 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:29.094949    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7" podUID="46881476-3103-4ee9-9b90-ec0ba9f30c3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 31 23:22:29.368464 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:29.358665    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x" event={"ID":"74a97819-25d1-43c8-b104-ee21999c5f68","Type":"ContainerStarted","Data":"b8a55f36c3f978f59a8e1afb3b32f9e273dd589f1308456c72eff537bbfac6c6"}
Oct 31 23:22:29.373800 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:29.373774    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p" event={"ID":"394e1575-7171-45a2-8d19-ef551eaad0ea","Type":"ContainerStarted","Data":"b17227da4c1379614a3ac17d6f7bc0e0c17b0b4b9f4fae005652472059b2797f"}
Oct 31 23:22:29.393837 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:29.389737    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62" event={"ID":"d05b75f2-18af-4d86-99d9-a51d2d23c3b5","Type":"ContainerStarted","Data":"920f66b2cd028172d5b338559802c2525925afcdb8927e7363df4d657c4a1107"}
Oct 31 23:22:29.407282 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:29.406525    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb1cddef-0805-49a6-86dc-4489565710bd" path="/var/lib/kubelet/pods/bb1cddef-0805-49a6-86dc-4489565710bd/volumes"
Oct 31 23:22:29.407282 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:29.406971    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27" path="/var/lib/kubelet/pods/cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27/volumes"
Oct 31 23:22:29.414898 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:29.414862    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:22:29.747436 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:29.746426    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:22:29.760386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:29.755717    4301 patch_prober.go:29] interesting pod/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 23:22:29.760386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:29.755781    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62" podUID="d05b75f2-18af-4d86-99d9-a51d2d23c3b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 23:22:30.035457 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:30.017332    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:22:30.035457 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:30.018991    4301 patch_prober.go:29] interesting pod/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 31 23:22:30.035457 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:30.019046    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x" podUID="74a97819-25d1-43c8-b104-ee21999c5f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 31 23:22:30.111406 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:30.111337    4301 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 31 23:22:30.111710 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:30.111681    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7" podUID="46881476-3103-4ee9-9b90-ec0ba9f30c3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 31 23:22:30.288706 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:30.287704    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:22:30.289281 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:30.288966    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 31 23:22:30.289281 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:30.289021    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p" podUID="394e1575-7171-45a2-8d19-ef551eaad0ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 31 23:22:30.757394 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:30.756057    4301 patch_prober.go:29] interesting pod/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 23:22:30.757394 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:30.756114    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62" podUID="d05b75f2-18af-4d86-99d9-a51d2d23c3b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 23:22:31.019192 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:31.014769    4301 patch_prober.go:29] interesting pod/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 31 23:22:31.019192 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:31.014834    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x" podUID="74a97819-25d1-43c8-b104-ee21999c5f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 31 23:22:31.068388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:31.067449    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:31.091384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:31.088863    4301 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 31 23:22:31.091384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:31.088931    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7" podUID="46881476-3103-4ee9-9b90-ec0ba9f30c3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 31 23:22:31.272463 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:31.271467    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:31.295382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:31.289802    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 31 23:22:31.295382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:31.289862    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p" podUID="394e1575-7171-45a2-8d19-ef551eaad0ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 31 23:22:31.777205 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:31.723331    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97" podUID="c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:31.777205 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:31.734175    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 31 23:22:31.777205 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:31.752122    4301 patch_prober.go:29] interesting pod/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 23:22:31.777205 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:31.752182    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62" podUID="d05b75f2-18af-4d86-99d9-a51d2d23c3b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 23:22:31.777205 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:31.772482    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:32.056350 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:31.941540    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:32.056350 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:31.953216    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 23:22:32.056350 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:31.979527    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 23:22:32.135535 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:32.079287    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97" podUID="c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:32.185441 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:32.140478    4301 patch_prober.go:29] interesting pod/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 31 23:22:32.185441 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:32.140539    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x" podUID="74a97819-25d1-43c8-b104-ee21999c5f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 31 23:22:32.202478 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:32.200426    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:32.382785 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:32.298592    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:32.406205 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:32.398247    4301 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 31 23:22:32.406205 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:32.398309    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7" podUID="46881476-3103-4ee9-9b90-ec0ba9f30c3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 31 23:22:32.492384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:32.465735    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 31 23:22:32.508543 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:32.508489    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p" podUID="394e1575-7171-45a2-8d19-ef551eaad0ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 31 23:22:32.783463 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:32.762418    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:32.783463 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:32.762790    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97" podUID="c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:32.783463 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:32.763386    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 31 23:22:32.810427 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:32.795529    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:32.851604 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:22:32.851566    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:22:32.923073 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:22:32.886046    4301 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.436s"
Oct 31 23:22:32.923073 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:32.886301    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"
Oct 31 23:22:32.923073 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:32.886332    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4528ed-64b678979d-spcpj"]
Oct 31 23:22:32.923073 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:32.886398    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4528ed-5446b78459-n8x8d"]
Oct 31 23:22:32.923073 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:32.886565    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:22:32.923073 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:32.887049    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:22:32.985581 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:32.985541    4301 patch_prober.go:29] interesting pod/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 23:22:32.985863 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:32.985834    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62" podUID="d05b75f2-18af-4d86-99d9-a51d2d23c3b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 23:22:33.043010 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:33.042917    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:33.149284 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:33.149249    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:22:33.165588 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:33.043431    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 23:22:33.193257 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:33.193210    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:22:33.202387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:33.043838    4301 patch_prober.go:29] interesting pod/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 31 23:22:33.297443 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:33.291433    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x" podUID="74a97819-25d1-43c8-b104-ee21999c5f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 31 23:22:33.322611 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:33.210500    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:33.322988 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:33.043460    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 23:22:33.350815 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:33.350787    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:22:33.351155 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:33.351141    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:22:33.418464 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:33.418417    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:33.495761 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:33.493829    4301 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 31 23:22:33.495761 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:33.493909    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7" podUID="46881476-3103-4ee9-9b90-ec0ba9f30c3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 31 23:22:33.608465 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:33.607861    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:33.627832 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:33.626790    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 31 23:22:33.627832 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:33.627209    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p" podUID="394e1575-7171-45a2-8d19-ef551eaad0ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 31 23:22:33.770841 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:33.770106    4301 patch_prober.go:29] interesting pod/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 23:22:33.770841 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:33.770176    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62" podUID="d05b75f2-18af-4d86-99d9-a51d2d23c3b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 23:22:33.891735 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:33.891601    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 31 23:22:33.899552 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:33.891666    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97" podUID="c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 31 23:22:33.929590 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:33.929558    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:22:33.929922 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:33.929906    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:22:33.954332 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:33.954292    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:34.054059 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:34.052831    4301 patch_prober.go:29] interesting pod/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 31 23:22:34.054059 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:34.052890    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x" podUID="74a97819-25d1-43c8-b104-ee21999c5f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 31 23:22:34.157458 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:34.157343    4301 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 31 23:22:34.157757 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:34.157726    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7" podUID="46881476-3103-4ee9-9b90-ec0ba9f30c3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 31 23:22:34.309005 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:34.200588    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:34.309005 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:34.202130    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:34.312777 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:34.229759    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:34.315961 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:34.315932    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 31 23:22:34.316177 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:34.316152    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p" podUID="394e1575-7171-45a2-8d19-ef551eaad0ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 31 23:22:34.323845 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:34.323796    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 23:22:34.326773 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:34.326742    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:34.415454 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:34.402884    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"]
Oct 31 23:22:34.438588 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:34.438558    4301 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 23:22:34.464112 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:34.429102    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 31 23:22:34.464295 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:34.464170    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:22:34.752524 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:34.752432    4301 patch_prober.go:29] interesting pod/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 23:22:34.784382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:34.780706    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:34.785134 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:34.753517    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62" podUID="d05b75f2-18af-4d86-99d9-a51d2d23c3b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 23:22:34.903594 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:34.903549    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97" podUID="c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:34.942540 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:34.942491    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:34.984585 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:34.967223    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:35.079484 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:35.056309    4301 patch_prober.go:29] interesting pod/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 31 23:22:35.079484 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:35.056419    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x" podUID="74a97819-25d1-43c8-b104-ee21999c5f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 31 23:22:35.202180 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:35.126332    4301 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 31 23:22:35.202180 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:35.126411    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7" podUID="46881476-3103-4ee9-9b90-ec0ba9f30c3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 31 23:22:35.325380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:35.264480    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 23:22:35.325380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:35.265940    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 23:22:35.325380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:35.282930    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:22:35.325380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:35.303674    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 31 23:22:35.325380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:35.303724    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p" podUID="394e1575-7171-45a2-8d19-ef551eaad0ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 31 23:22:35.325380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:35.316460    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:35.370434 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:35.326293    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:35.370434 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:35.327734    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:35.479064 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:35.464512    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:35.510384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:35.495049    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:35.808462 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:35.764297    4301 patch_prober.go:29] interesting pod/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 23:22:35.808462 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:35.764377    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62" podUID="d05b75f2-18af-4d86-99d9-a51d2d23c3b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 23:22:35.808462 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:35.791504    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:35.893767 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:35.893733    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:22:35.957462 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:35.914832    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97" podUID="c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 31 23:22:35.959337 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:35.946695    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:36.019208 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:36.017443    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:36.174315 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:36.112339    4301 patch_prober.go:29] interesting pod/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 31 23:22:36.174315 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:36.173482    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x" podUID="74a97819-25d1-43c8-b104-ee21999c5f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 31 23:22:36.247008 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:36.225547    4301 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 31 23:22:36.247008 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:36.225616    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7" podUID="46881476-3103-4ee9-9b90-ec0ba9f30c3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 31 23:22:36.262976 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:36.261857    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:22:36.285411 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:36.270784    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 23:22:36.285411 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:36.275286    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:36.301606 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:36.298921    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 31 23:22:36.301606 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:36.298993    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p" podUID="394e1575-7171-45a2-8d19-ef551eaad0ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 31 23:22:36.332470 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:36.329847    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:36.332470 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:36.330225    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 23:22:36.332470 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:36.331701    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:36.526438 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:36.526066    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:22:36.768754 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:36.762332    4301 patch_prober.go:29] interesting pod/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 23:22:36.768754 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:36.762415    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62" podUID="d05b75f2-18af-4d86-99d9-a51d2d23c3b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 23:22:36.768754 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:36.766659    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 23:22:36.768754 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:36.766728    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:36.861680 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:36.861571    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 31 23:22:36.909591 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:36.897731    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:37.007392 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:37.005569    4301 patch_prober.go:29] interesting pod/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 31 23:22:37.007392 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:37.005653    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x" podUID="74a97819-25d1-43c8-b104-ee21999c5f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 31 23:22:37.007392 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:37.006327    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:22:37.041295 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:37.040231    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:37.067567 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:37.067521    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97" podUID="c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:37.333398 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:37.253399    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:37.333398 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:37.324602    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 31 23:22:37.333398 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:37.324725    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p" podUID="394e1575-7171-45a2-8d19-ef551eaad0ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 31 23:22:37.333398 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:37.327075    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 23:22:37.333398 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:37.328267    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Oct 31 23:22:37.333398 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:37.328520    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:37.410223 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:37.410156    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:22:37.415715 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:37.412925    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 23:22:37.471796 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:37.459804    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:37.537797 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:37.537753    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:37.619440 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:37.609501    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:37.823380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:37.819648    4301 patch_prober.go:29] interesting pod/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 23:22:37.823380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:37.819718    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62" podUID="d05b75f2-18af-4d86-99d9-a51d2d23c3b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 23:22:37.865378 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:37.864602    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:37.910301 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:37.910162    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-4528ed-64b678979d-spcpj" podUID="a3756c42-084c-4986-84f3-720fcf5ad4e1" containerName="payload-processing" containerID="cri-o://4d60c7c010310bb505d3b83f26b669716fbace36a44c9c1b2eabe37f7eec344a" gracePeriod=25
Oct 31 23:22:37.913588 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:37.913552    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4528ed-5446b78459-n8x8d" podUID="9ec0f289-97b7-45f0-952c-309c3b97d414" containerName="payload-pre-processing" containerID="cri-o://f6fb7fed841e3be3031a82bad57f808919bec5be3a0e37bdf65a4799c580cece" gracePeriod=25
Oct 31 23:22:38.015436 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.014830    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:22:38.015436 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.014892    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:22:38.015436 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.014918    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:22:38.019862 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.018686    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:38.026379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.023717    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:22:38.026379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.024881    4301 patch_prober.go:29] interesting pod/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 31 23:22:38.026379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.024926    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x" podUID="74a97819-25d1-43c8-b104-ee21999c5f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 31 23:22:38.047384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.030840    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:22:38.047384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.032036    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"
Oct 31 23:22:38.138886 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.121842    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:22:38.138886 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.130325    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:22:38.203152 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.202415    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:22:38.241012 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.240774    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:22:38.294829 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.294696    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:22:38.296200 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.296127    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:22:38.304457 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.298603    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:22:38.649823 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.640080    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4528ed-5446b78459-n8x8d"
Oct 31 23:22:38.720515 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.718052    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4528ed-64b678979d-spcpj"
Oct 31 23:22:38.749982 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.749948    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:22:38.753956 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.752674    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ec0f289-97b7-45f0-952c-309c3b97d414-kube-api-access-fk4tp\" (UniqueName: \"kubernetes.io/projected/9ec0f289-97b7-45f0-952c-309c3b97d414-kube-api-access-fk4tp\") pod \"9ec0f289-97b7-45f0-952c-309c3b97d414\" (UID: \"9ec0f289-97b7-45f0-952c-309c3b97d414\") "
Oct 31 23:22:38.753956 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.752806    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ec0f289-97b7-45f0-952c-309c3b97d414-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ec0f289-97b7-45f0-952c-309c3b97d414-plugins-config-volume\") pod \"9ec0f289-97b7-45f0-952c-309c3b97d414\" (UID: \"9ec0f289-97b7-45f0-952c-309c3b97d414\") "
Oct 31 23:22:38.753956 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.752925    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3756c42-084c-4986-84f3-720fcf5ad4e1-kube-api-access-b6cqv\" (UniqueName: \"kubernetes.io/projected/a3756c42-084c-4986-84f3-720fcf5ad4e1-kube-api-access-b6cqv\") pod \"a3756c42-084c-4986-84f3-720fcf5ad4e1\" (UID: \"a3756c42-084c-4986-84f3-720fcf5ad4e1\") "
Oct 31 23:22:38.753956 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.753012    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3756c42-084c-4986-84f3-720fcf5ad4e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3756c42-084c-4986-84f3-720fcf5ad4e1-plugins-config-volume\") pod \"a3756c42-084c-4986-84f3-720fcf5ad4e1\" (UID: \"a3756c42-084c-4986-84f3-720fcf5ad4e1\") "
Oct 31 23:22:38.754374 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.754015    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ec0f289-97b7-45f0-952c-309c3b97d414-plugins-config-volume" pod "9ec0f289-97b7-45f0-952c-309c3b97d414" (UID: "9ec0f289-97b7-45f0-952c-309c3b97d414"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:22:38.754729 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.754615    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3756c42-084c-4986-84f3-720fcf5ad4e1-plugins-config-volume" pod "a3756c42-084c-4986-84f3-720fcf5ad4e1" (UID: "a3756c42-084c-4986-84f3-720fcf5ad4e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:22:38.762534 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.762498    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3756c42-084c-4986-84f3-720fcf5ad4e1-kube-api-access-b6cqv" pod "a3756c42-084c-4986-84f3-720fcf5ad4e1" (UID: "a3756c42-084c-4986-84f3-720fcf5ad4e1"). InnerVolumeSpecName "kube-api-access-b6cqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:22:38.762957 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.762933    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec0f289-97b7-45f0-952c-309c3b97d414-kube-api-access-fk4tp" pod "9ec0f289-97b7-45f0-952c-309c3b97d414" (UID: "9ec0f289-97b7-45f0-952c-309c3b97d414"). InnerVolumeSpecName "kube-api-access-fk4tp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:22:38.856019 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.855938    4301 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ec0f289-97b7-45f0-952c-309c3b97d414-plugins-config-volume\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:38.856019 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.855973    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6cqv\" (UniqueName: \"kubernetes.io/projected/a3756c42-084c-4986-84f3-720fcf5ad4e1-kube-api-access-b6cqv\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:38.856019 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.855986    4301 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3756c42-084c-4986-84f3-720fcf5ad4e1-plugins-config-volume\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:38.856019 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.855998    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk4tp\" (UniqueName: \"kubernetes.io/projected/9ec0f289-97b7-45f0-952c-309c3b97d414-kube-api-access-fk4tp\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:38.882372 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.882322    4301 generic.go:356] "Generic (PLEG): container finished" podID="a3756c42-084c-4986-84f3-720fcf5ad4e1" containerID="4d60c7c010310bb505d3b83f26b669716fbace36a44c9c1b2eabe37f7eec344a" exitCode=0
Oct 31 23:22:38.882557 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.882385    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4528ed-64b678979d-spcpj"
Oct 31 23:22:38.882557 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.882392    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4528ed-64b678979d-spcpj" event={"ID":"a3756c42-084c-4986-84f3-720fcf5ad4e1","Type":"ContainerDied","Data":"4d60c7c010310bb505d3b83f26b669716fbace36a44c9c1b2eabe37f7eec344a"}
Oct 31 23:22:38.882557 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.882463    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4528ed-64b678979d-spcpj" event={"ID":"a3756c42-084c-4986-84f3-720fcf5ad4e1","Type":"ContainerDied","Data":"bb5590d41dccdb9b4617c1502f6eba05b947a1471f8325e456d11a026d79fd30"}
Oct 31 23:22:38.882557 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.882495    4301 scope.go:122] "RemoveContainer" containerID="4d60c7c010310bb505d3b83f26b669716fbace36a44c9c1b2eabe37f7eec344a"
Oct 31 23:22:38.884198 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.884171    4301 generic.go:356] "Generic (PLEG): container finished" podID="9ec0f289-97b7-45f0-952c-309c3b97d414" containerID="f6fb7fed841e3be3031a82bad57f808919bec5be3a0e37bdf65a4799c580cece" exitCode=0
Oct 31 23:22:38.884338 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.884316    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4528ed-5446b78459-n8x8d" event={"ID":"9ec0f289-97b7-45f0-952c-309c3b97d414","Type":"ContainerDied","Data":"f6fb7fed841e3be3031a82bad57f808919bec5be3a0e37bdf65a4799c580cece"}
Oct 31 23:22:38.884462 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.884442    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4528ed-5446b78459-n8x8d" event={"ID":"9ec0f289-97b7-45f0-952c-309c3b97d414","Type":"ContainerDied","Data":"ea2cb1e6efc0d8af219b1b449823a6a8ef1da2e2c034181d75d2d658933a13e1"}
Oct 31 23:22:38.884587 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.884572    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4528ed-5446b78459-n8x8d"
Oct 31 23:22:38.886333 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.886313    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:22:38.897737 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.897713    4301 scope.go:122] "RemoveContainer" containerID="4d60c7c010310bb505d3b83f26b669716fbace36a44c9c1b2eabe37f7eec344a"
Oct 31 23:22:38.898096 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:22:38.898070    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d60c7c010310bb505d3b83f26b669716fbace36a44c9c1b2eabe37f7eec344a\": container with ID starting with 4d60c7c010310bb505d3b83f26b669716fbace36a44c9c1b2eabe37f7eec344a not found: ID does not exist" containerID="4d60c7c010310bb505d3b83f26b669716fbace36a44c9c1b2eabe37f7eec344a"
Oct 31 23:22:38.898196 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.898108    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d60c7c010310bb505d3b83f26b669716fbace36a44c9c1b2eabe37f7eec344a"} err="failed to get container status \"4d60c7c010310bb505d3b83f26b669716fbace36a44c9c1b2eabe37f7eec344a\": rpc error: code = NotFound desc = could not find container \"4d60c7c010310bb505d3b83f26b669716fbace36a44c9c1b2eabe37f7eec344a\": container with ID starting with 4d60c7c010310bb505d3b83f26b669716fbace36a44c9c1b2eabe37f7eec344a not found: ID does not exist"
Oct 31 23:22:38.898196 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.898132    4301 scope.go:122] "RemoveContainer" containerID="f6fb7fed841e3be3031a82bad57f808919bec5be3a0e37bdf65a4799c580cece"
Oct 31 23:22:38.908146 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.908129    4301 scope.go:122] "RemoveContainer" containerID="f6fb7fed841e3be3031a82bad57f808919bec5be3a0e37bdf65a4799c580cece"
Oct 31 23:22:38.908489 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:22:38.908464    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6fb7fed841e3be3031a82bad57f808919bec5be3a0e37bdf65a4799c580cece\": container with ID starting with f6fb7fed841e3be3031a82bad57f808919bec5be3a0e37bdf65a4799c580cece not found: ID does not exist" containerID="f6fb7fed841e3be3031a82bad57f808919bec5be3a0e37bdf65a4799c580cece"
Oct 31 23:22:38.908552 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.908501    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6fb7fed841e3be3031a82bad57f808919bec5be3a0e37bdf65a4799c580cece"} err="failed to get container status \"f6fb7fed841e3be3031a82bad57f808919bec5be3a0e37bdf65a4799c580cece\": rpc error: code = NotFound desc = could not find container \"f6fb7fed841e3be3031a82bad57f808919bec5be3a0e37bdf65a4799c580cece\": container with ID starting with f6fb7fed841e3be3031a82bad57f808919bec5be3a0e37bdf65a4799c580cece not found: ID does not exist"
Oct 31 23:22:38.940060 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.940026    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4528ed-64b678979d-spcpj"]
Oct 31 23:22:38.944577 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.944550    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4528ed-64b678979d-spcpj"]
Oct 31 23:22:38.958890 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.958858    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4528ed-5446b78459-n8x8d"]
Oct 31 23:22:38.971677 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:38.971624    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4528ed-5446b78459-n8x8d"]
Oct 31 23:22:39.011682 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:39.010460    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:22:39.012257 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:39.012230    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:22:39.480469 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:39.474764    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ec0f289-97b7-45f0-952c-309c3b97d414" path="/var/lib/kubelet/pods/9ec0f289-97b7-45f0-952c-309c3b97d414/volumes"
Oct 31 23:22:39.480469 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:39.475465    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3756c42-084c-4986-84f3-720fcf5ad4e1" path="/var/lib/kubelet/pods/a3756c42-084c-4986-84f3-720fcf5ad4e1/volumes"
Oct 31 23:22:42.898097 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:42.898055    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"]
Oct 31 23:22:42.899061 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:42.899036    4301 cpu_manager.go:414] "containerMap: removing container" podUID="bb1cddef-0805-49a6-86dc-4489565710bd" containerName="payload-pre-processing"
Oct 31 23:22:42.899061 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:42.899059    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb1cddef-0805-49a6-86dc-4489565710bd" containerName="payload-pre-processing"
Oct 31 23:22:42.899242 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:42.899077    4301 cpu_manager.go:414] "containerMap: removing container" podUID="53842309-02b2-4ce4-a9e6-334334c57b84" containerName="revoke-keys"
Oct 31 23:22:42.899242 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:42.899085    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53842309-02b2-4ce4-a9e6-334334c57b84" containerName="revoke-keys"
Oct 31 23:22:42.899242 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:42.899099    4301 cpu_manager.go:414] "containerMap: removing container" podUID="893091bf-0eb6-4443-8060-b35d05383b05" containerName="revoke-keys"
Oct 31 23:22:42.899242 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:42.899107    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="893091bf-0eb6-4443-8060-b35d05383b05" containerName="revoke-keys"
Oct 31 23:22:42.899242 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:42.899142    4301 cpu_manager.go:414] "containerMap: removing container" podUID="cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27" containerName="payload-processing"
Oct 31 23:22:42.899242 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:42.899150    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27" containerName="payload-processing"
Oct 31 23:22:42.899242 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:42.899185    4301 cpu_manager.go:414] "containerMap: removing container" podUID="a3756c42-084c-4986-84f3-720fcf5ad4e1" containerName="payload-processing"
Oct 31 23:22:42.899242 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:42.899192    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3756c42-084c-4986-84f3-720fcf5ad4e1" containerName="payload-processing"
Oct 31 23:22:42.899242 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:42.899202    4301 cpu_manager.go:414] "containerMap: removing container" podUID="9ec0f289-97b7-45f0-952c-309c3b97d414" containerName="payload-pre-processing"
Oct 31 23:22:42.899242 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:42.899209    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ec0f289-97b7-45f0-952c-309c3b97d414" containerName="payload-pre-processing"
Oct 31 23:22:42.899759 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:42.899320    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="cabc6ea0-3cd9-4a11-bb51-d674b3dd7a27" containerName="payload-processing"
Oct 31 23:22:42.899759 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:42.899335    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3756c42-084c-4986-84f3-720fcf5ad4e1" containerName="payload-processing"
Oct 31 23:22:42.899759 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:42.899344    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ec0f289-97b7-45f0-952c-309c3b97d414" containerName="payload-pre-processing"
Oct 31 23:22:42.899759 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:42.899372    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb1cddef-0805-49a6-86dc-4489565710bd" containerName="payload-pre-processing"
Oct 31 23:22:42.899759 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:42.899383    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="53842309-02b2-4ce4-a9e6-334334c57b84" containerName="revoke-keys"
Oct 31 23:22:42.899759 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:42.899393    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="893091bf-0eb6-4443-8060-b35d05383b05" containerName="revoke-keys"
Oct 31 23:22:42.905286 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:42.905059    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:22:42.934026 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:42.933993    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"]
Oct 31 23:22:42.979836 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:42.979802    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a05c486-5fda-45d2-95f7-57a8779a34a6-istiod-ca-cert\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:22:42.980010 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:42.979848    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntd6n\" (UniqueName: \"kubernetes.io/projected/4a05c486-5fda-45d2-95f7-57a8779a34a6-kube-api-access-ntd6n\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:22:42.980010 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:42.979938    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-workload-socket\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:22:42.980010 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:42.979963    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a05c486-5fda-45d2-95f7-57a8779a34a6-istio-token\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:22:42.980010 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:42.979982    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-workload-certs\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:22:42.980272 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:42.980080    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-istio-envoy\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:22:42.980272 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:42.980119    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-istio-data\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:22:42.980272 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:42.980168    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a05c486-5fda-45d2-95f7-57a8779a34a6-istio-podinfo\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:22:42.980272 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:42.980251    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-credential-socket\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:22:43.081726 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.081667    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-credential-socket\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:22:43.084157 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.081884    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a05c486-5fda-45d2-95f7-57a8779a34a6-istiod-ca-cert\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:22:43.084157 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.081933    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntd6n\" (UniqueName: \"kubernetes.io/projected/4a05c486-5fda-45d2-95f7-57a8779a34a6-kube-api-access-ntd6n\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:22:43.084157 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.081990    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-workload-socket\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:22:43.084157 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.082023    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-credential-socket\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:22:43.084157 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.082126    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a05c486-5fda-45d2-95f7-57a8779a34a6-istio-token\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:22:43.084157 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.082163    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-workload-certs\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:22:43.084157 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.082218    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-istio-envoy\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:22:43.084157 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.082256    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-istio-data\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:22:43.084157 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.082308    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-workload-socket\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:22:43.084157 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.082381    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a05c486-5fda-45d2-95f7-57a8779a34a6-istio-podinfo\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:22:43.084157 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.082607    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-workload-certs\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:22:43.084157 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.082684    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-istio-data\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:22:43.084157 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.082699    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a05c486-5fda-45d2-95f7-57a8779a34a6-istiod-ca-cert\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:22:43.084933 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.084528    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-istio-envoy\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:22:43.084933 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.084858    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a05c486-5fda-45d2-95f7-57a8779a34a6-istio-podinfo\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:22:43.102284 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.102249    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntd6n\" (UniqueName: \"kubernetes.io/projected/4a05c486-5fda-45d2-95f7-57a8779a34a6-kube-api-access-ntd6n\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:22:43.102738 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.102717    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a05c486-5fda-45d2-95f7-57a8779a34a6-istio-token\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:22:43.195255 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.195161    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"]
Oct 31 23:22:43.200970 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.200944    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:22:43.208633 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.208607    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-bf279c-openshift-default-dockercfg-bstvr\""
Oct 31 23:22:43.216748 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.216719    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"]
Oct 31 23:22:43.240421 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.238150    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:22:43.287577 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.287487    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-workload-certs\") pod \"e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") " pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:22:43.288486 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.288461    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-workload-socket\") pod \"e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") " pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:22:43.288949 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.288929    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8844\" (UniqueName: \"kubernetes.io/projected/e9de1723-3c15-4c53-8080-26e27ac118e4-kube-api-access-b8844\") pod \"e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") " pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:22:43.289147 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.289131    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-istio-envoy\") pod \"e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") " pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:22:43.289331 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.289316    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-istio-data\") pod \"e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") " pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:22:43.290147 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.290129    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9de1723-3c15-4c53-8080-26e27ac118e4-istiod-ca-cert\") pod \"e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") " pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:22:43.290523 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.290504    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9de1723-3c15-4c53-8080-26e27ac118e4-istio-podinfo\") pod \"e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") " pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:22:43.319664 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.319622    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-credential-socket\") pod \"e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") " pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:22:43.320014 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.319992    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9de1723-3c15-4c53-8080-26e27ac118e4-istio-token\") pod \"e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") " pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:22:43.491735 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.443492    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-workload-certs\") pod \"e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") " pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:22:43.491735 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.443580    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-workload-socket\") pod \"e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") " pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:22:43.491735 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.443751    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8844\" (UniqueName: \"kubernetes.io/projected/e9de1723-3c15-4c53-8080-26e27ac118e4-kube-api-access-b8844\") pod \"e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") " pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:22:43.491735 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.443800    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-istio-envoy\") pod \"e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") " pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:22:43.491735 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.443853    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-istio-data\") pod \"e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") " pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:22:43.491735 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.443889    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9de1723-3c15-4c53-8080-26e27ac118e4-istiod-ca-cert\") pod \"e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") " pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:22:43.491735 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.444006    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9de1723-3c15-4c53-8080-26e27ac118e4-istio-podinfo\") pod \"e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") " pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:22:43.491735 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.444081    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-credential-socket\") pod \"e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") " pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:22:43.491735 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.444127    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9de1723-3c15-4c53-8080-26e27ac118e4-istio-token\") pod \"e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") " pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:22:43.491735 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.444982    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-workload-certs\") pod \"e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") " pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:22:43.491735 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.465238    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-credential-socket\") pod \"e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") " pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:22:43.491735 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.465507    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-workload-socket\") pod \"e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") " pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:22:43.516810 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.500767    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-istio-data\") pod \"e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") " pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:22:43.516810 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.501840    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-istio-envoy\") pod \"e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") " pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:22:43.534383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.530515    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9de1723-3c15-4c53-8080-26e27ac118e4-istio-podinfo\") pod \"e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") " pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:22:43.650168 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.650133    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9de1723-3c15-4c53-8080-26e27ac118e4-istiod-ca-cert\") pod \"e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") " pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:22:43.729023 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.726089    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9de1723-3c15-4c53-8080-26e27ac118e4-istio-token\") pod \"e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") " pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:22:43.785428 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.783272    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8844\" (UniqueName: \"kubernetes.io/projected/e9de1723-3c15-4c53-8080-26e27ac118e4-kube-api-access-b8844\") pod \"e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") " pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:22:43.825895 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:43.825578    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:22:44.092548 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:44.092514    4301 generic.go:356] "Generic (PLEG): container finished" podID="f7560a60-cbba-43ba-81a1-946c57663b80" containerID="aa732f12d6a9ce2ff5978da02c04bf10f8256e93d9e297a2cbd2d997864ada20" exitCode=0
Oct 31 23:22:44.092955 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:44.092721    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" event={"ID":"f7560a60-cbba-43ba-81a1-946c57663b80","Type":"ContainerDied","Data":"aa732f12d6a9ce2ff5978da02c04bf10f8256e93d9e297a2cbd2d997864ada20"}
Oct 31 23:22:44.092955 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:44.092786    4301 scope.go:122] "RemoveContainer" containerID="b711c656e8708f7a548a7453b46176951c1341be574dfd1c54c987141351ef13"
Oct 31 23:22:44.093322 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:44.093301    4301 scope.go:122] "RemoveContainer" containerID="aa732f12d6a9ce2ff5978da02c04bf10f8256e93d9e297a2cbd2d997864ada20"
Oct 31 23:22:44.097657 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:44.097607    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:22:44.097918 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:44.097890    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:22:44.098110 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:44.098088    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:22:44.278739 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:44.277345    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"]
Oct 31 23:22:44.295608 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:44.293679    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:22:44.295608 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:44.293767    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:22:44.295608 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:44.293810    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:22:44.297437 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:22:44.297162    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a05c486_5fda_45d2_95f7_57a8779a34a6.slice/crio-42236602974be242d4aa5a317d032d6cf74d0da5484066510cf5c24803dad643 WatchSource:0}: Error finding container 42236602974be242d4aa5a317d032d6cf74d0da5484066510cf5c24803dad643: Status 404 returned error can't find the container with id 42236602974be242d4aa5a317d032d6cf74d0da5484066510cf5c24803dad643
Oct 31 23:22:44.331852 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:44.331815    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"]
Oct 31 23:22:44.353959 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:44.344073    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:22:44.353959 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:44.344157    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:22:44.353959 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:44.344197    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:22:45.103449 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:45.103411    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99" event={"ID":"4a05c486-5fda-45d2-95f7-57a8779a34a6","Type":"ContainerStarted","Data":"693875dcdf09637209b08e5a73905fd483501e815d9549fb15b01df72fc7a381"}
Oct 31 23:22:45.103889 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:45.103479    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99" event={"ID":"4a05c486-5fda-45d2-95f7-57a8779a34a6","Type":"ContainerStarted","Data":"42236602974be242d4aa5a317d032d6cf74d0da5484066510cf5c24803dad643"}
Oct 31 23:22:45.105275 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:45.105243    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds" event={"ID":"e9de1723-3c15-4c53-8080-26e27ac118e4","Type":"ContainerStarted","Data":"b5a8fc8b2e0385dddba02178f1b292286a99189e348c99bf202d5738c63fbb6a"}
Oct 31 23:22:45.105419 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:45.105287    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds" event={"ID":"e9de1723-3c15-4c53-8080-26e27ac118e4","Type":"ContainerStarted","Data":"e3f8e7b62bec2e690418d58e011344ab9660e4fab49e0ec4fba877cd729bfd81"}
Oct 31 23:22:45.107653 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:45.107634    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" event={"ID":"f7560a60-cbba-43ba-81a1-946c57663b80","Type":"ContainerStarted","Data":"6b48c06c7ecfdd2a87f194966f50cd3ba912dcfd514ea4cf071f07ce8a6ef305"}
Oct 31 23:22:45.129153 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:45.129098    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99" podStartSLOduration=3.129084801 podStartE2EDuration="3.129084801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:22:45.126627422 +0000 UTC m=+1983.873816929" watchObservedRunningTime="2026-10-31 23:22:45.129084801 +0000 UTC m=+1983.876274307"
Oct 31 23:22:45.194184 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:45.194127    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds" podStartSLOduration=2.194111797 podStartE2EDuration="2.194111797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:22:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:22:45.191422439 +0000 UTC m=+1983.938611943" watchObservedRunningTime="2026-10-31 23:22:45.194111797 +0000 UTC m=+1983.941301301"
Oct 31 23:22:45.239083 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:45.239048    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:22:45.241156 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:45.241126    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 23:22:45.241306 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:45.241177    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99" podUID="4a05c486-5fda-45d2-95f7-57a8779a34a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 23:22:45.844387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:45.835896    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:22:45.844387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:45.842348    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:22:45.844387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:45.842494    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:22:45.844387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:45.842535    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:22:45.867382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:45.864296    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:22:46.115473 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:46.115387    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:22:46.116263 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:46.116240    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:22:46.248388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:46.240804    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 23:22:46.248388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:46.240865    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99" podUID="4a05c486-5fda-45d2-95f7-57a8779a34a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 23:22:46.838971 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:46.838922    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:22:46.839300 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:46.839270    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:22:47.244884 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:47.244840    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 23:22:47.245538 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:47.245505    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99" podUID="4a05c486-5fda-45d2-95f7-57a8779a34a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 23:22:47.849383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:47.841803    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:22:47.849383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:47.841889    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:22:48.256389 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:48.251440    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 23:22:48.256389 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:48.251501    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99" podUID="4a05c486-5fda-45d2-95f7-57a8779a34a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 23:22:48.534443 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:48.533769    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"]
Oct 31 23:22:48.534443 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:48.534013    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz" podUID="b0f47424-1492-437c-9da2-0712ea4266b5" containerName="istio-proxy" containerID="cri-o://b3d2e931eb8f500c51cb822f0538e89808c092ca3f05650b780f8554704d49d0" gracePeriod=30
Oct 31 23:22:48.841624 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:48.841519    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:22:48.841936 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:48.841898    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:22:48.896437 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:48.888681    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:22:48.935381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:48.934608    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"]
Oct 31 23:22:48.935381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:48.934976    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g" podUID="5d3c8af1-7e81-43b7-998c-6ba875936720" containerName="istio-proxy" containerID="cri-o://05caa5d59190e2005628db6cec4e139e099aa7abadc4edc23a3f3c201fc9725e" gracePeriod=30
Oct 31 23:22:49.082455 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:49.082416    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"]
Oct 31 23:22:49.082924 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:49.082892    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln" podUID="94a46479-0793-4c0a-8d40-b342c343899c" containerName="istio-proxy" containerID="cri-o://06b0ca79d22ed7dba1e69c509321737c3f64bb6eac466ce2916510c61b88ec14" gracePeriod=30
Oct 31 23:22:49.254382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:49.253810    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 23:22:49.254382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:49.253885    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99" podUID="4a05c486-5fda-45d2-95f7-57a8779a34a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 23:22:49.290444 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:49.290407    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bf279c-8d7bdc44b-j7pcc"]
Oct 31 23:22:49.301594 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:49.300310    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bf279c-8d7bdc44b-j7pcc"
Oct 31 23:22:49.301594 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:49.300221    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bf279c-8d7bdc44b-j7pcc"]
Oct 31 23:22:49.395880 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:49.395855    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-bf279c\""
Oct 31 23:22:49.398498 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:49.398471    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-bf279c-dockercfg-54tb6\""
Oct 31 23:22:49.402898 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:49.402875    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8594\" (UniqueName: \"kubernetes.io/projected/63efe5e5-d71d-41e1-a359-d8d13471f32b-kube-api-access-z8594\") pod \"payload-processing-e2e-api-b-bf279c-8d7bdc44b-j7pcc\" (UID: \"63efe5e5-d71d-41e1-a359-d8d13471f32b\") " pod="openshift-ingress/payload-processing-e2e-api-b-bf279c-8d7bdc44b-j7pcc"
Oct 31 23:22:49.403184 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:49.403162    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63efe5e5-d71d-41e1-a359-d8d13471f32b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bf279c-8d7bdc44b-j7pcc\" (UID: \"63efe5e5-d71d-41e1-a359-d8d13471f32b\") " pod="openshift-ingress/payload-processing-e2e-api-b-bf279c-8d7bdc44b-j7pcc"
Oct 31 23:22:49.508015 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:49.507921    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8594\" (UniqueName: \"kubernetes.io/projected/63efe5e5-d71d-41e1-a359-d8d13471f32b-kube-api-access-z8594\") pod \"payload-processing-e2e-api-b-bf279c-8d7bdc44b-j7pcc\" (UID: \"63efe5e5-d71d-41e1-a359-d8d13471f32b\") " pod="openshift-ingress/payload-processing-e2e-api-b-bf279c-8d7bdc44b-j7pcc"
Oct 31 23:22:49.523962 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:49.523929    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63efe5e5-d71d-41e1-a359-d8d13471f32b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bf279c-8d7bdc44b-j7pcc\" (UID: \"63efe5e5-d71d-41e1-a359-d8d13471f32b\") " pod="openshift-ingress/payload-processing-e2e-api-b-bf279c-8d7bdc44b-j7pcc"
Oct 31 23:22:49.542418 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:49.538968    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63efe5e5-d71d-41e1-a359-d8d13471f32b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bf279c-8d7bdc44b-j7pcc\" (UID: \"63efe5e5-d71d-41e1-a359-d8d13471f32b\") " pod="openshift-ingress/payload-processing-e2e-api-b-bf279c-8d7bdc44b-j7pcc"
Oct 31 23:22:49.574530 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:49.561150    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8594\" (UniqueName: \"kubernetes.io/projected/63efe5e5-d71d-41e1-a359-d8d13471f32b-kube-api-access-z8594\") pod \"payload-processing-e2e-api-b-bf279c-8d7bdc44b-j7pcc\" (UID: \"63efe5e5-d71d-41e1-a359-d8d13471f32b\") " pod="openshift-ingress/payload-processing-e2e-api-b-bf279c-8d7bdc44b-j7pcc"
Oct 31 23:22:49.632100 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:49.632068    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bf279c-8d7bdc44b-j7pcc"
Oct 31 23:22:49.841291 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:49.841253    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:22:49.841659 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:49.841599    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:22:50.378384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:50.360078    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bf279c-8d7bdc44b-j7pcc"]
Oct 31 23:22:50.418102 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:50.418066    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 23:22:50.418342 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:50.418315    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99" podUID="4a05c486-5fda-45d2-95f7-57a8779a34a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 23:22:50.492287 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:50.492243    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:22:50.671405 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:50.666082    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:22:50.842578 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:50.839911    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:22:50.842578 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:50.839970    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:22:51.244250 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:51.244205    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 23:22:51.244588 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:51.244554    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99" podUID="4a05c486-5fda-45d2-95f7-57a8779a34a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 23:22:51.282932 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:51.282890    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bf279c-8d7bdc44b-j7pcc" event={"ID":"63efe5e5-d71d-41e1-a359-d8d13471f32b","Type":"ContainerStarted","Data":"0f8773480712a972cad285d95319cb75993ace445fc193bb593478e0fe694751"}
Oct 31 23:22:51.283228 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:51.283201    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bf279c-8d7bdc44b-j7pcc" event={"ID":"63efe5e5-d71d-41e1-a359-d8d13471f32b","Type":"ContainerStarted","Data":"cb413acd712e6a629f359a8bf71deaa0b55533d68f6c7a2396f787815a972f41"}
Oct 31 23:22:51.297394 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:51.286488    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-bf279c-8d7bdc44b-j7pcc"
Oct 31 23:22:51.308180 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:51.308143    4301 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 23:22:51.309487 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:51.309452    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln" podUID="94a46479-0793-4c0a-8d40-b342c343899c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 23:22:51.323981 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:51.322573    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-bf279c-8d7bdc44b-j7pcc" podStartSLOduration=2.322552405 podStartE2EDuration="2.322552405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:22:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:22:51.320766966 +0000 UTC m=+1990.067956504" watchObservedRunningTime="2026-10-31 23:22:51.322552405 +0000 UTC m=+1990.069741911"
Oct 31 23:22:51.895381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:51.891134    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4eff5f-2qsml"]
Oct 31 23:22:51.950519 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:51.950483    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:22:51.950807 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:51.950779    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:22:51.973433 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:51.952846    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4eff5f-2qsml"
Oct 31 23:22:51.993036 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:51.984083    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 23:22:51.993036 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:51.984278    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\""
Oct 31 23:22:52.012981 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:52.012896    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4eff5f-2qsml"]
Oct 31 23:22:52.053551 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:52.053506    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af80a837-3fb4-4f97-bd01-0268b934397f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4eff5f-2qsml\" (UID: \"af80a837-3fb4-4f97-bd01-0268b934397f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4eff5f-2qsml"
Oct 31 23:22:52.054229 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:52.054193    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5jgq\" (UniqueName: \"kubernetes.io/projected/af80a837-3fb4-4f97-bd01-0268b934397f-kube-api-access-n5jgq\") pod \"maas-api-revoke-keys-e2e-api-a-4eff5f-2qsml\" (UID: \"af80a837-3fb4-4f97-bd01-0268b934397f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4eff5f-2qsml"
Oct 31 23:22:52.054490 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:52.054468    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af80a837-3fb4-4f97-bd01-0268b934397f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4eff5f-2qsml\" (UID: \"af80a837-3fb4-4f97-bd01-0268b934397f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4eff5f-2qsml"
Oct 31 23:22:52.173439 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:52.162309    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5jgq\" (UniqueName: \"kubernetes.io/projected/af80a837-3fb4-4f97-bd01-0268b934397f-kube-api-access-n5jgq\") pod \"maas-api-revoke-keys-e2e-api-a-4eff5f-2qsml\" (UID: \"af80a837-3fb4-4f97-bd01-0268b934397f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4eff5f-2qsml"
Oct 31 23:22:52.173439 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:52.162509    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af80a837-3fb4-4f97-bd01-0268b934397f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4eff5f-2qsml\" (UID: \"af80a837-3fb4-4f97-bd01-0268b934397f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4eff5f-2qsml"
Oct 31 23:22:52.173439 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:52.169585    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af80a837-3fb4-4f97-bd01-0268b934397f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4eff5f-2qsml\" (UID: \"af80a837-3fb4-4f97-bd01-0268b934397f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4eff5f-2qsml"
Oct 31 23:22:52.175345 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:52.175286    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af80a837-3fb4-4f97-bd01-0268b934397f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4eff5f-2qsml\" (UID: \"af80a837-3fb4-4f97-bd01-0268b934397f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4eff5f-2qsml"
Oct 31 23:22:52.200212 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:52.200177    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af80a837-3fb4-4f97-bd01-0268b934397f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4eff5f-2qsml\" (UID: \"af80a837-3fb4-4f97-bd01-0268b934397f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4eff5f-2qsml"
Oct 31 23:22:52.224549 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:52.224465    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5jgq\" (UniqueName: \"kubernetes.io/projected/af80a837-3fb4-4f97-bd01-0268b934397f-kube-api-access-n5jgq\") pod \"maas-api-revoke-keys-e2e-api-a-4eff5f-2qsml\" (UID: \"af80a837-3fb4-4f97-bd01-0268b934397f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4eff5f-2qsml"
Oct 31 23:22:52.270417 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:52.260400    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 23:22:52.270417 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:52.260477    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99" podUID="4a05c486-5fda-45d2-95f7-57a8779a34a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 23:22:52.331475 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:52.331159    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4eff5f-2qsml"
Oct 31 23:22:52.702103 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:52.693930    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97" podUID="c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" containerName="main" containerID="cri-o://2fd72b7805b68199659be0e62969013b8900b097fe1fd251420d82d982d313ed" gracePeriod=42
Oct 31 23:22:52.897395 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:52.847508    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 23:22:52.897395 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:52.847568    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 23:22:52.909136 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:52.907899    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97" podUID="c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": EOF"
Oct 31 23:22:52.981715 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:52.975539    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97" podUID="c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": dial tcp 10.133.0.97:8000: connect: connection refused"
Oct 31 23:22:52.981715 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:52.975664    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"
Oct 31 23:22:53.189811 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.189774    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4eff5f-2qsml"]
Oct 31 23:22:53.291535 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.289461    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 23:22:53.291535 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.289519    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99" podUID="4a05c486-5fda-45d2-95f7-57a8779a34a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 23:22:53.342411 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.341661    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"
Oct 31 23:22:53.402031 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.401999    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-tmp-dir\") pod \"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0\" (UID: \"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0\") "
Oct 31 23:22:53.402255 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.402078    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-dshm\") pod \"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0\" (UID: \"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0\") "
Oct 31 23:22:53.402255 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.402231    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-model-cache\") pod \"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0\" (UID: \"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0\") "
Oct 31 23:22:53.402394 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.402307    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-kube-api-access-zlxx9\" (UniqueName: \"kubernetes.io/projected/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-kube-api-access-zlxx9\") pod \"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0\" (UID: \"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0\") "
Oct 31 23:22:53.402394 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.402333    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-tmp-dir" pod "c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" (UID: "c2f530bc-ac7a-4ce3-96f1-fdddb435ace0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:53.402509 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.402381    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-home\" (UniqueName: \"kubernetes.io/empty-dir/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-home\") pod \"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0\" (UID: \"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0\") "
Oct 31 23:22:53.402509 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.402486    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-model-cache" pod "c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" (UID: "c2f530bc-ac7a-4ce3-96f1-fdddb435ace0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:53.402509 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.402496    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-tls-certs\" (UniqueName: \"kubernetes.io/secret/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-tls-certs\") pod \"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0\" (UID: \"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0\") "
Oct 31 23:22:53.402670 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.402582    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-home" pod "c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" (UID: "c2f530bc-ac7a-4ce3-96f1-fdddb435ace0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:53.404179 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.404150    4301 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-tmp-dir\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:53.404179 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.404179    4301 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-model-cache\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:53.404417 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.404195    4301 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-home\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:53.404920 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.404895    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-kube-api-access-zlxx9" pod "c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" (UID: "c2f530bc-ac7a-4ce3-96f1-fdddb435ace0"). InnerVolumeSpecName "kube-api-access-zlxx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:22:53.405196 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.405171    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-dshm" pod "c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" (UID: "c2f530bc-ac7a-4ce3-96f1-fdddb435ace0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:53.405549 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.405527    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-tls-certs" pod "c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" (UID: "c2f530bc-ac7a-4ce3-96f1-fdddb435ace0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:22:53.471007 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.470972    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4eff5f-2qsml" event={"ID":"af80a837-3fb4-4f97-bd01-0268b934397f","Type":"ContainerStarted","Data":"f37ca3358c550b0349202f86ed266a7ed6e6006caa772810c1d53d382573d97c"}
Oct 31 23:22:53.484911 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.484872    4301 generic.go:356] "Generic (PLEG): container finished" podID="c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" containerID="2fd72b7805b68199659be0e62969013b8900b097fe1fd251420d82d982d313ed" exitCode=0
Oct 31 23:22:53.485081 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.484950    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97" event={"ID":"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0","Type":"ContainerDied","Data":"2fd72b7805b68199659be0e62969013b8900b097fe1fd251420d82d982d313ed"}
Oct 31 23:22:53.485081 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.484986    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97" event={"ID":"c2f530bc-ac7a-4ce3-96f1-fdddb435ace0","Type":"ContainerDied","Data":"3a4c9624581e2ad717a2326c97e10bef08f734a2d742863c944d7ad3cfd1fc68"}
Oct 31 23:22:53.485081 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.485007    4301 scope.go:122] "RemoveContainer" containerID="2fd72b7805b68199659be0e62969013b8900b097fe1fd251420d82d982d313ed"
Oct 31 23:22:53.486503 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.485326    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"
Oct 31 23:22:53.502056 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.502027    4301 scope.go:122] "RemoveContainer" containerID="2fd72b7805b68199659be0e62969013b8900b097fe1fd251420d82d982d313ed"
Oct 31 23:22:53.502401 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:22:53.502374    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fd72b7805b68199659be0e62969013b8900b097fe1fd251420d82d982d313ed\": container with ID starting with 2fd72b7805b68199659be0e62969013b8900b097fe1fd251420d82d982d313ed not found: ID does not exist" containerID="2fd72b7805b68199659be0e62969013b8900b097fe1fd251420d82d982d313ed"
Oct 31 23:22:53.502519 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.502405    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fd72b7805b68199659be0e62969013b8900b097fe1fd251420d82d982d313ed"} err="failed to get container status \"2fd72b7805b68199659be0e62969013b8900b097fe1fd251420d82d982d313ed\": rpc error: code = NotFound desc = could not find container \"2fd72b7805b68199659be0e62969013b8900b097fe1fd251420d82d982d313ed\": container with ID starting with 2fd72b7805b68199659be0e62969013b8900b097fe1fd251420d82d982d313ed not found: ID does not exist"
Oct 31 23:22:53.505643 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.505609    4301 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-tls-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:53.505643 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.505643    4301 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-dshm\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:53.505787 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.505654    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlxx9\" (UniqueName: \"kubernetes.io/projected/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0-kube-api-access-zlxx9\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:53.516044 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.516013    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"]
Oct 31 23:22:53.526268 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.526237    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-36a15fc5/e2e-lifecycle-model-36a15fc5-kserve-657697c645-lcb97"]
Oct 31 23:22:53.892496 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.892462    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:22:53.894667 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:53.894642    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:22:54.030767 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.030687    4301 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 23:22:54.031287 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.031254    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz" podUID="b0f47424-1492-437c-9da2-0712ea4266b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 23:22:54.254945 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.254912    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:22:54.270320 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.270284    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:22:54.272757 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.272730    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:22:54.478853 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.478501    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:22:54.526421 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.503387    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-istio-envoy\") pod \"b0f47424-1492-437c-9da2-0712ea4266b5\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") "
Oct 31 23:22:54.526421 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.503522    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0f47424-1492-437c-9da2-0712ea4266b5-kube-api-access-plp92\" (UniqueName: \"kubernetes.io/projected/b0f47424-1492-437c-9da2-0712ea4266b5-kube-api-access-plp92\") pod \"b0f47424-1492-437c-9da2-0712ea4266b5\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") "
Oct 31 23:22:54.526421 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.503568    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-istio-data\") pod \"b0f47424-1492-437c-9da2-0712ea4266b5\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") "
Oct 31 23:22:54.526421 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.503597    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0f47424-1492-437c-9da2-0712ea4266b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0f47424-1492-437c-9da2-0712ea4266b5-istiod-ca-cert\") pod \"b0f47424-1492-437c-9da2-0712ea4266b5\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") "
Oct 31 23:22:54.526421 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.503674    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0f47424-1492-437c-9da2-0712ea4266b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0f47424-1492-437c-9da2-0712ea4266b5-istio-podinfo\") pod \"b0f47424-1492-437c-9da2-0712ea4266b5\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") "
Oct 31 23:22:54.526421 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.503796    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-credential-socket\") pod \"b0f47424-1492-437c-9da2-0712ea4266b5\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") "
Oct 31 23:22:54.526421 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.503829    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0f47424-1492-437c-9da2-0712ea4266b5-istio-token\" (UniqueName: \"kubernetes.io/projected/b0f47424-1492-437c-9da2-0712ea4266b5-istio-token\") pod \"b0f47424-1492-437c-9da2-0712ea4266b5\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") "
Oct 31 23:22:54.526421 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.503859    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-workload-socket\") pod \"b0f47424-1492-437c-9da2-0712ea4266b5\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") "
Oct 31 23:22:54.526421 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.503952    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-workload-certs\") pod \"b0f47424-1492-437c-9da2-0712ea4266b5\" (UID: \"b0f47424-1492-437c-9da2-0712ea4266b5\") "
Oct 31 23:22:54.526421 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.504902    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-workload-certs" pod "b0f47424-1492-437c-9da2-0712ea4266b5" (UID: "b0f47424-1492-437c-9da2-0712ea4266b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:54.526421 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.509863    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-credential-socket" pod "b0f47424-1492-437c-9da2-0712ea4266b5" (UID: "b0f47424-1492-437c-9da2-0712ea4266b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:54.526421 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.512811    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-istio-data" pod "b0f47424-1492-437c-9da2-0712ea4266b5" (UID: "b0f47424-1492-437c-9da2-0712ea4266b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:54.526421 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.513101    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0f47424-1492-437c-9da2-0712ea4266b5-istiod-ca-cert" pod "b0f47424-1492-437c-9da2-0712ea4266b5" (UID: "b0f47424-1492-437c-9da2-0712ea4266b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:22:54.526421 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.514930    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-workload-socket" pod "b0f47424-1492-437c-9da2-0712ea4266b5" (UID: "b0f47424-1492-437c-9da2-0712ea4266b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:54.526421 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.518704    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-istio-envoy" pod "b0f47424-1492-437c-9da2-0712ea4266b5" (UID: "b0f47424-1492-437c-9da2-0712ea4266b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:54.551283 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.550738    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0f47424-1492-437c-9da2-0712ea4266b5-istio-podinfo" pod "b0f47424-1492-437c-9da2-0712ea4266b5" (UID: "b0f47424-1492-437c-9da2-0712ea4266b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:22:54.551283 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.550779    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0f47424-1492-437c-9da2-0712ea4266b5-istio-token" pod "b0f47424-1492-437c-9da2-0712ea4266b5" (UID: "b0f47424-1492-437c-9da2-0712ea4266b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:22:54.554383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.552146    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0f47424-1492-437c-9da2-0712ea4266b5-kube-api-access-plp92" pod "b0f47424-1492-437c-9da2-0712ea4266b5" (UID: "b0f47424-1492-437c-9da2-0712ea4266b5"). InnerVolumeSpecName "kube-api-access-plp92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:22:54.556577 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.556550    4301 generic.go:356] "Generic (PLEG): container finished" podID="b0f47424-1492-437c-9da2-0712ea4266b5" containerID="b3d2e931eb8f500c51cb822f0538e89808c092ca3f05650b780f8554704d49d0" exitCode=0
Oct 31 23:22:54.556860 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.556839    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz" event={"ID":"b0f47424-1492-437c-9da2-0712ea4266b5","Type":"ContainerDied","Data":"b3d2e931eb8f500c51cb822f0538e89808c092ca3f05650b780f8554704d49d0"}
Oct 31 23:22:54.557060 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.557041    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz" event={"ID":"b0f47424-1492-437c-9da2-0712ea4266b5","Type":"ContainerDied","Data":"e5a527a5dcfcf8acb78ce13059e226554f02dfb340abdebede69501683766342"}
Oct 31 23:22:54.557173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.557155    4301 scope.go:122] "RemoveContainer" containerID="b3d2e931eb8f500c51cb822f0538e89808c092ca3f05650b780f8554704d49d0"
Oct 31 23:22:54.557468 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.557348    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"
Oct 31 23:22:54.558569 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.558232    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:22:54.579156 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.567166    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4eff5f-2qsml" event={"ID":"af80a837-3fb4-4f97-bd01-0268b934397f","Type":"ContainerStarted","Data":"31c5f1148655812fc527f4593d8a06e1a69ff0768487851fead68f144e676e97"}
Oct 31 23:22:54.598325 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.597156    4301 scope.go:122] "RemoveContainer" containerID="b3d2e931eb8f500c51cb822f0538e89808c092ca3f05650b780f8554704d49d0"
Oct 31 23:22:54.598325 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:22:54.597911    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3d2e931eb8f500c51cb822f0538e89808c092ca3f05650b780f8554704d49d0\": container with ID starting with b3d2e931eb8f500c51cb822f0538e89808c092ca3f05650b780f8554704d49d0 not found: ID does not exist" containerID="b3d2e931eb8f500c51cb822f0538e89808c092ca3f05650b780f8554704d49d0"
Oct 31 23:22:54.598325 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.597950    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3d2e931eb8f500c51cb822f0538e89808c092ca3f05650b780f8554704d49d0"} err="failed to get container status \"b3d2e931eb8f500c51cb822f0538e89808c092ca3f05650b780f8554704d49d0\": rpc error: code = NotFound desc = could not find container \"b3d2e931eb8f500c51cb822f0538e89808c092ca3f05650b780f8554704d49d0\": container with ID starting with b3d2e931eb8f500c51cb822f0538e89808c092ca3f05650b780f8554704d49d0 not found: ID does not exist"
Oct 31 23:22:54.599397 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.598892    4301 generic.go:356] "Generic (PLEG): container finished" podID="94a46479-0793-4c0a-8d40-b342c343899c" containerID="06b0ca79d22ed7dba1e69c509321737c3f64bb6eac466ce2916510c61b88ec14" exitCode=0
Oct 31 23:22:54.599397 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.598967    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln" event={"ID":"94a46479-0793-4c0a-8d40-b342c343899c","Type":"ContainerDied","Data":"06b0ca79d22ed7dba1e69c509321737c3f64bb6eac466ce2916510c61b88ec14"}
Oct 31 23:22:54.606313 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.606282    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d3c8af1-7e81-43b7-998c-6ba875936720-kube-api-access-sp7cr\" (UniqueName: \"kubernetes.io/projected/5d3c8af1-7e81-43b7-998c-6ba875936720-kube-api-access-sp7cr\") pod \"5d3c8af1-7e81-43b7-998c-6ba875936720\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") "
Oct 31 23:22:54.607042 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.606964    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-istio-envoy\") pod \"5d3c8af1-7e81-43b7-998c-6ba875936720\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") "
Oct 31 23:22:54.613195 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.613168    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d3c8af1-7e81-43b7-998c-6ba875936720-istio-token\" (UniqueName: \"kubernetes.io/projected/5d3c8af1-7e81-43b7-998c-6ba875936720-istio-token\") pod \"5d3c8af1-7e81-43b7-998c-6ba875936720\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") "
Oct 31 23:22:54.641734 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.641688    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d3c8af1-7e81-43b7-998c-6ba875936720-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d3c8af1-7e81-43b7-998c-6ba875936720-istiod-ca-cert\") pod \"5d3c8af1-7e81-43b7-998c-6ba875936720\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") "
Oct 31 23:22:54.642130 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.642094    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-workload-socket\") pod \"5d3c8af1-7e81-43b7-998c-6ba875936720\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") "
Oct 31 23:22:54.642473 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.642449    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-istio-data\") pod \"5d3c8af1-7e81-43b7-998c-6ba875936720\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") "
Oct 31 23:22:54.642664 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.642649    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d3c8af1-7e81-43b7-998c-6ba875936720-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d3c8af1-7e81-43b7-998c-6ba875936720-istio-podinfo\") pod \"5d3c8af1-7e81-43b7-998c-6ba875936720\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") "
Oct 31 23:22:54.642833 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.642819    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-credential-socket\") pod \"5d3c8af1-7e81-43b7-998c-6ba875936720\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") "
Oct 31 23:22:54.643064 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.643048    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-workload-certs\") pod \"5d3c8af1-7e81-43b7-998c-6ba875936720\" (UID: \"5d3c8af1-7e81-43b7-998c-6ba875936720\") "
Oct 31 23:22:54.644219 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.644183    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:54.644351 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.644337    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0f47424-1492-437c-9da2-0712ea4266b5-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:54.644502 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.644488    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:54.644616 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.644604    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:54.644727 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.644714    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:54.644826 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.644815    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plp92\" (UniqueName: \"kubernetes.io/projected/b0f47424-1492-437c-9da2-0712ea4266b5-kube-api-access-plp92\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:54.644913 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.644902    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0f47424-1492-437c-9da2-0712ea4266b5-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:54.644992 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.644980    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0f47424-1492-437c-9da2-0712ea4266b5-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:54.645073 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.645063    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0f47424-1492-437c-9da2-0712ea4266b5-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:54.645140 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.636459    4301 generic.go:356] "Generic (PLEG): container finished" podID="5d3c8af1-7e81-43b7-998c-6ba875936720" containerID="05caa5d59190e2005628db6cec4e139e099aa7abadc4edc23a3f3c201fc9725e" exitCode=0
Oct 31 23:22:54.645945 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.636485    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g" event={"ID":"5d3c8af1-7e81-43b7-998c-6ba875936720","Type":"ContainerDied","Data":"05caa5d59190e2005628db6cec4e139e099aa7abadc4edc23a3f3c201fc9725e"}
Oct 31 23:22:54.646048 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.645994    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g" event={"ID":"5d3c8af1-7e81-43b7-998c-6ba875936720","Type":"ContainerDied","Data":"4af14cf85ab5edbf21137c2e6f7bc304551ba089300b837c70fe0bd853a22cbe"}
Oct 31 23:22:54.646048 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.646035    4301 scope.go:122] "RemoveContainer" containerID="05caa5d59190e2005628db6cec4e139e099aa7abadc4edc23a3f3c201fc9725e"
Oct 31 23:22:54.646375 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.638673    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"
Oct 31 23:22:54.647315 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.644656    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d3c8af1-7e81-43b7-998c-6ba875936720-istiod-ca-cert" pod "5d3c8af1-7e81-43b7-998c-6ba875936720" (UID: "5d3c8af1-7e81-43b7-998c-6ba875936720"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:22:54.647442 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.645913    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-credential-socket" pod "5d3c8af1-7e81-43b7-998c-6ba875936720" (UID: "5d3c8af1-7e81-43b7-998c-6ba875936720"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:54.647535 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.646331    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-workload-socket" pod "5d3c8af1-7e81-43b7-998c-6ba875936720" (UID: "5d3c8af1-7e81-43b7-998c-6ba875936720"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:54.647612 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.646657    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-workload-certs" pod "5d3c8af1-7e81-43b7-998c-6ba875936720" (UID: "5d3c8af1-7e81-43b7-998c-6ba875936720"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:54.647683 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.646726    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-istio-data" pod "5d3c8af1-7e81-43b7-998c-6ba875936720" (UID: "5d3c8af1-7e81-43b7-998c-6ba875936720"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:54.652308 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.652237    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4eff5f-2qsml" podStartSLOduration=3.652219541 podStartE2EDuration="3.652219541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:22:54.641293144 +0000 UTC m=+1993.388482654" watchObservedRunningTime="2026-10-31 23:22:54.652219541 +0000 UTC m=+1993.399409047"
Oct 31 23:22:54.653025 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.652988    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d3c8af1-7e81-43b7-998c-6ba875936720-istio-podinfo" pod "5d3c8af1-7e81-43b7-998c-6ba875936720" (UID: "5d3c8af1-7e81-43b7-998c-6ba875936720"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:22:54.653851 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.653816    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-istio-envoy" pod "5d3c8af1-7e81-43b7-998c-6ba875936720" (UID: "5d3c8af1-7e81-43b7-998c-6ba875936720"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:54.655999 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.655931    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d3c8af1-7e81-43b7-998c-6ba875936720-kube-api-access-sp7cr" pod "5d3c8af1-7e81-43b7-998c-6ba875936720" (UID: "5d3c8af1-7e81-43b7-998c-6ba875936720"). InnerVolumeSpecName "kube-api-access-sp7cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:22:54.657963 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.657162    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d3c8af1-7e81-43b7-998c-6ba875936720-istio-token" pod "5d3c8af1-7e81-43b7-998c-6ba875936720" (UID: "5d3c8af1-7e81-43b7-998c-6ba875936720"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:22:54.671208 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.670900    4301 scope.go:122] "RemoveContainer" containerID="05caa5d59190e2005628db6cec4e139e099aa7abadc4edc23a3f3c201fc9725e"
Oct 31 23:22:54.671891 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:22:54.671795    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05caa5d59190e2005628db6cec4e139e099aa7abadc4edc23a3f3c201fc9725e\": container with ID starting with 05caa5d59190e2005628db6cec4e139e099aa7abadc4edc23a3f3c201fc9725e not found: ID does not exist" containerID="05caa5d59190e2005628db6cec4e139e099aa7abadc4edc23a3f3c201fc9725e"
Oct 31 23:22:54.671891 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.671829    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05caa5d59190e2005628db6cec4e139e099aa7abadc4edc23a3f3c201fc9725e"} err="failed to get container status \"05caa5d59190e2005628db6cec4e139e099aa7abadc4edc23a3f3c201fc9725e\": rpc error: code = NotFound desc = could not find container \"05caa5d59190e2005628db6cec4e139e099aa7abadc4edc23a3f3c201fc9725e\": container with ID starting with 05caa5d59190e2005628db6cec4e139e099aa7abadc4edc23a3f3c201fc9725e not found: ID does not exist"
Oct 31 23:22:54.672538 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.672515    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"]
Oct 31 23:22:54.678886 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.678859    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-2cfbz"]
Oct 31 23:22:54.747206 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.747169    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:54.747413 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.747221    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d3c8af1-7e81-43b7-998c-6ba875936720-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:54.747413 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.747245    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d3c8af1-7e81-43b7-998c-6ba875936720-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:54.747413 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.747264    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:54.747413 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.747282    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:54.747413 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.747296    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d3c8af1-7e81-43b7-998c-6ba875936720-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:54.747413 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.747310    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:54.747413 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.747324    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d3c8af1-7e81-43b7-998c-6ba875936720-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:54.747413 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.747335    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp7cr\" (UniqueName: \"kubernetes.io/projected/5d3c8af1-7e81-43b7-998c-6ba875936720-kube-api-access-sp7cr\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:54.754264 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.754237    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:22:54.848160 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.848059    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-istio-envoy\") pod \"94a46479-0793-4c0a-8d40-b342c343899c\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") "
Oct 31 23:22:54.848341 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.848175    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94a46479-0793-4c0a-8d40-b342c343899c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94a46479-0793-4c0a-8d40-b342c343899c-istio-podinfo\") pod \"94a46479-0793-4c0a-8d40-b342c343899c\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") "
Oct 31 23:22:54.848341 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.848222    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94a46479-0793-4c0a-8d40-b342c343899c-istio-token\" (UniqueName: \"kubernetes.io/projected/94a46479-0793-4c0a-8d40-b342c343899c-istio-token\") pod \"94a46479-0793-4c0a-8d40-b342c343899c\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") "
Oct 31 23:22:54.848341 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.848279    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-credential-socket\") pod \"94a46479-0793-4c0a-8d40-b342c343899c\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") "
Oct 31 23:22:54.848584 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.848339    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-istio-data\") pod \"94a46479-0793-4c0a-8d40-b342c343899c\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") "
Oct 31 23:22:54.848584 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.848463    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-workload-certs\") pod \"94a46479-0793-4c0a-8d40-b342c343899c\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") "
Oct 31 23:22:54.848584 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.848535    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-workload-socket\") pod \"94a46479-0793-4c0a-8d40-b342c343899c\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") "
Oct 31 23:22:54.848758 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.848609    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94a46479-0793-4c0a-8d40-b342c343899c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94a46479-0793-4c0a-8d40-b342c343899c-istiod-ca-cert\") pod \"94a46479-0793-4c0a-8d40-b342c343899c\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") "
Oct 31 23:22:54.848810 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.848786    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-istio-data" pod "94a46479-0793-4c0a-8d40-b342c343899c" (UID: "94a46479-0793-4c0a-8d40-b342c343899c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:54.848972 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.848899    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-credential-socket" pod "94a46479-0793-4c0a-8d40-b342c343899c" (UID: "94a46479-0793-4c0a-8d40-b342c343899c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:54.849689 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.849099    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-workload-certs" pod "94a46479-0793-4c0a-8d40-b342c343899c" (UID: "94a46479-0793-4c0a-8d40-b342c343899c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:54.849689 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.848975    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-workload-socket" pod "94a46479-0793-4c0a-8d40-b342c343899c" (UID: "94a46479-0793-4c0a-8d40-b342c343899c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:54.849689 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.849226    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94a46479-0793-4c0a-8d40-b342c343899c-istiod-ca-cert" pod "94a46479-0793-4c0a-8d40-b342c343899c" (UID: "94a46479-0793-4c0a-8d40-b342c343899c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:22:54.850085 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.850050    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94a46479-0793-4c0a-8d40-b342c343899c-kube-api-access-6fv5f\" (UniqueName: \"kubernetes.io/projected/94a46479-0793-4c0a-8d40-b342c343899c-kube-api-access-6fv5f\") pod \"94a46479-0793-4c0a-8d40-b342c343899c\" (UID: \"94a46479-0793-4c0a-8d40-b342c343899c\") "
Oct 31 23:22:54.850962 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.850937    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:54.851065 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.850966    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:54.851065 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.850980    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:54.851065 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.850993    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:54.851065 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.851007    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94a46479-0793-4c0a-8d40-b342c343899c-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:54.851617 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.851597    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-istio-envoy" pod "94a46479-0793-4c0a-8d40-b342c343899c" (UID: "94a46479-0793-4c0a-8d40-b342c343899c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:22:54.851725 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.851657    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a46479-0793-4c0a-8d40-b342c343899c-istio-token" pod "94a46479-0793-4c0a-8d40-b342c343899c" (UID: "94a46479-0793-4c0a-8d40-b342c343899c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:22:54.851725 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.851717    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94a46479-0793-4c0a-8d40-b342c343899c-istio-podinfo" pod "94a46479-0793-4c0a-8d40-b342c343899c" (UID: "94a46479-0793-4c0a-8d40-b342c343899c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:22:54.853190 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.853166    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a46479-0793-4c0a-8d40-b342c343899c-kube-api-access-6fv5f" pod "94a46479-0793-4c0a-8d40-b342c343899c" (UID: "94a46479-0793-4c0a-8d40-b342c343899c"). InnerVolumeSpecName "kube-api-access-6fv5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:22:54.952633 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.952538    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fv5f\" (UniqueName: \"kubernetes.io/projected/94a46479-0793-4c0a-8d40-b342c343899c-kube-api-access-6fv5f\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:54.952633 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.952573    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94a46479-0793-4c0a-8d40-b342c343899c-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:54.952633 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.952585    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94a46479-0793-4c0a-8d40-b342c343899c-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:54.952633 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.952607    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94a46479-0793-4c0a-8d40-b342c343899c-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:54.977907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.977874    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"]
Oct 31 23:22:54.982892 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:54.982863    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-gf26g"]
Oct 31 23:22:55.406156 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:55.406124    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d3c8af1-7e81-43b7-998c-6ba875936720" path="/var/lib/kubelet/pods/5d3c8af1-7e81-43b7-998c-6ba875936720/volumes"
Oct 31 23:22:55.406725 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:55.406709    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0f47424-1492-437c-9da2-0712ea4266b5" path="/var/lib/kubelet/pods/b0f47424-1492-437c-9da2-0712ea4266b5/volumes"
Oct 31 23:22:55.407151 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:55.407136    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" path="/var/lib/kubelet/pods/c2f530bc-ac7a-4ce3-96f1-fdddb435ace0/volumes"
Oct 31 23:22:55.655875 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:55.655840    4301 generic.go:356] "Generic (PLEG): container finished" podID="af80a837-3fb4-4f97-bd01-0268b934397f" containerID="31c5f1148655812fc527f4593d8a06e1a69ff0768487851fead68f144e676e97" exitCode=0
Oct 31 23:22:55.656073 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:55.655927    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4eff5f-2qsml" event={"ID":"af80a837-3fb4-4f97-bd01-0268b934397f","Type":"ContainerDied","Data":"31c5f1148655812fc527f4593d8a06e1a69ff0768487851fead68f144e676e97"}
Oct 31 23:22:55.657436 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:55.657351    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"
Oct 31 23:22:55.657436 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:55.657387    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln" event={"ID":"94a46479-0793-4c0a-8d40-b342c343899c","Type":"ContainerDied","Data":"72c25d292a3c0c69df2a258f0a79ae1e0f3e877de33df70245d767b28e06262a"}
Oct 31 23:22:55.657701 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:55.657447    4301 scope.go:122] "RemoveContainer" containerID="06b0ca79d22ed7dba1e69c509321737c3f64bb6eac466ce2916510c61b88ec14"
Oct 31 23:22:55.691815 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:55.691785    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"]
Oct 31 23:22:55.700864 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:55.700830    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-9v7ln"]
Oct 31 23:22:56.994390 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:56.992432    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4528ed-4hkf5"]
Oct 31 23:22:57.006384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:57.005497    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4528ed-4hkf5"]
Oct 31 23:22:57.111389 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:57.109317    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4eff5f-2qsml"
Oct 31 23:22:57.257441 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:57.237144    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af80a837-3fb4-4f97-bd01-0268b934397f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af80a837-3fb4-4f97-bd01-0268b934397f-maas-api-openshift-ca\") pod \"af80a837-3fb4-4f97-bd01-0268b934397f\" (UID: \"af80a837-3fb4-4f97-bd01-0268b934397f\") "
Oct 31 23:22:57.257441 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:57.237212    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af80a837-3fb4-4f97-bd01-0268b934397f-kube-api-access-n5jgq\" (UniqueName: \"kubernetes.io/projected/af80a837-3fb4-4f97-bd01-0268b934397f-kube-api-access-n5jgq\") pod \"af80a837-3fb4-4f97-bd01-0268b934397f\" (UID: \"af80a837-3fb4-4f97-bd01-0268b934397f\") "
Oct 31 23:22:57.257441 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:57.237310    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af80a837-3fb4-4f97-bd01-0268b934397f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af80a837-3fb4-4f97-bd01-0268b934397f-maas-api-xks-ca\") pod \"af80a837-3fb4-4f97-bd01-0268b934397f\" (UID: \"af80a837-3fb4-4f97-bd01-0268b934397f\") "
Oct 31 23:22:57.257441 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:57.253803    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af80a837-3fb4-4f97-bd01-0268b934397f-maas-api-openshift-ca" pod "af80a837-3fb4-4f97-bd01-0268b934397f" (UID: "af80a837-3fb4-4f97-bd01-0268b934397f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:22:57.353383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:57.346702    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af80a837-3fb4-4f97-bd01-0268b934397f-maas-api-openshift-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:57.487383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:57.484847    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af80a837-3fb4-4f97-bd01-0268b934397f-maas-api-xks-ca" pod "af80a837-3fb4-4f97-bd01-0268b934397f" (UID: "af80a837-3fb4-4f97-bd01-0268b934397f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:22:57.524677 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:57.522222    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af80a837-3fb4-4f97-bd01-0268b934397f-kube-api-access-n5jgq" pod "af80a837-3fb4-4f97-bd01-0268b934397f" (UID: "af80a837-3fb4-4f97-bd01-0268b934397f"). InnerVolumeSpecName "kube-api-access-n5jgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:22:57.531008 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:57.529886    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af80a837-3fb4-4f97-bd01-0268b934397f-kube-api-access-n5jgq\" (UniqueName: \"kubernetes.io/projected/af80a837-3fb4-4f97-bd01-0268b934397f-kube-api-access-n5jgq\") pod \"af80a837-3fb4-4f97-bd01-0268b934397f\" (UID: \"af80a837-3fb4-4f97-bd01-0268b934397f\") "
Oct 31 23:22:57.531008 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:57.530010    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af80a837-3fb4-4f97-bd01-0268b934397f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af80a837-3fb4-4f97-bd01-0268b934397f-maas-api-xks-ca\") pod \"af80a837-3fb4-4f97-bd01-0268b934397f\" (UID: \"af80a837-3fb4-4f97-bd01-0268b934397f\") "
Oct 31 23:22:57.531008 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:22:57.530977    4301 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/af80a837-3fb4-4f97-bd01-0268b934397f/volumes/kubernetes.io~secret/maas-api-xks-ca
Oct 31 23:22:57.531008 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:57.531000    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af80a837-3fb4-4f97-bd01-0268b934397f-maas-api-xks-ca" pod "af80a837-3fb4-4f97-bd01-0268b934397f" (UID: "af80a837-3fb4-4f97-bd01-0268b934397f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:22:57.531340 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:22:57.531067    4301 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/af80a837-3fb4-4f97-bd01-0268b934397f/volumes/kubernetes.io~projected/kube-api-access-n5jgq
Oct 31 23:22:57.531340 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:57.531118    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af80a837-3fb4-4f97-bd01-0268b934397f-kube-api-access-n5jgq" pod "af80a837-3fb4-4f97-bd01-0268b934397f" (UID: "af80a837-3fb4-4f97-bd01-0268b934397f"). InnerVolumeSpecName "kube-api-access-n5jgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:22:57.556258 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:57.556205    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="893091bf-0eb6-4443-8060-b35d05383b05" path="/var/lib/kubelet/pods/893091bf-0eb6-4443-8060-b35d05383b05/volumes"
Oct 31 23:22:57.557143 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:57.557120    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94a46479-0793-4c0a-8d40-b342c343899c" path="/var/lib/kubelet/pods/94a46479-0793-4c0a-8d40-b342c343899c/volumes"
Oct 31 23:22:57.632462 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:57.632426    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5jgq\" (UniqueName: \"kubernetes.io/projected/af80a837-3fb4-4f97-bd01-0268b934397f-kube-api-access-n5jgq\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:57.632462 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:57.632457    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af80a837-3fb4-4f97-bd01-0268b934397f-maas-api-xks-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:22:57.677194 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:57.677168    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4eff5f-2qsml"
Oct 31 23:22:57.677390 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:57.677205    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4eff5f-2qsml" event={"ID":"af80a837-3fb4-4f97-bd01-0268b934397f","Type":"ContainerDied","Data":"f37ca3358c550b0349202f86ed266a7ed6e6006caa772810c1d53d382573d97c"}
Oct 31 23:22:57.677390 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:22:57.677253    4301 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f37ca3358c550b0349202f86ed266a7ed6e6006caa772810c1d53d382573d97c"
Oct 31 23:23:02.489002 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:02.488967    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-bf279c-8d7bdc44b-j7pcc"
Oct 31 23:23:02.553515 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:23:02.553479    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:23:05.042381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.041137    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"]
Oct 31 23:23:05.042381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.041513    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" containerID="cri-o://6b48c06c7ecfdd2a87f194966f50cd3ba912dcfd514ea4cf071f07ce8a6ef305" gracePeriod=30
Oct 31 23:23:05.054020 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.053988    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"]
Oct 31 23:23:05.054307 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.054237    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99" podUID="4a05c486-5fda-45d2-95f7-57a8779a34a6" containerName="istio-proxy" containerID="cri-o://693875dcdf09637209b08e5a73905fd483501e815d9549fb15b01df72fc7a381" gracePeriod=30
Oct 31 23:23:05.626881 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.626850    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-885fdf-nxpx8"]
Oct 31 23:23:05.627988 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.627962    4301 cpu_manager.go:414] "containerMap: removing container" podUID="af80a837-3fb4-4f97-bd01-0268b934397f" containerName="revoke-keys"
Oct 31 23:23:05.628154 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.628141    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af80a837-3fb4-4f97-bd01-0268b934397f" containerName="revoke-keys"
Oct 31 23:23:05.628262 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.628251    4301 cpu_manager.go:414] "containerMap: removing container" podUID="c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" containerName="main"
Oct 31 23:23:05.628337 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.628327    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" containerName="main"
Oct 31 23:23:05.628488 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.628476    4301 cpu_manager.go:414] "containerMap: removing container" podUID="b0f47424-1492-437c-9da2-0712ea4266b5" containerName="istio-proxy"
Oct 31 23:23:05.628625 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.628611    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0f47424-1492-437c-9da2-0712ea4266b5" containerName="istio-proxy"
Oct 31 23:23:05.628782 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.628770    4301 cpu_manager.go:414] "containerMap: removing container" podUID="94a46479-0793-4c0a-8d40-b342c343899c" containerName="istio-proxy"
Oct 31 23:23:05.628869 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.628858    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94a46479-0793-4c0a-8d40-b342c343899c" containerName="istio-proxy"
Oct 31 23:23:05.629059 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.629045    4301 cpu_manager.go:414] "containerMap: removing container" podUID="5d3c8af1-7e81-43b7-998c-6ba875936720" containerName="istio-proxy"
Oct 31 23:23:05.629192 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.629179    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d3c8af1-7e81-43b7-998c-6ba875936720" containerName="istio-proxy"
Oct 31 23:23:05.629564 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.629547    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d3c8af1-7e81-43b7-998c-6ba875936720" containerName="istio-proxy"
Oct 31 23:23:05.629690 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.629680    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="af80a837-3fb4-4f97-bd01-0268b934397f" containerName="revoke-keys"
Oct 31 23:23:05.629779 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.629770    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2f530bc-ac7a-4ce3-96f1-fdddb435ace0" containerName="main"
Oct 31 23:23:05.629861 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.629851    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0f47424-1492-437c-9da2-0712ea4266b5" containerName="istio-proxy"
Oct 31 23:23:05.629933 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.629925    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="94a46479-0793-4c0a-8d40-b342c343899c" containerName="istio-proxy"
Oct 31 23:23:05.635720 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.635698    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-885fdf-nxpx8"
Oct 31 23:23:05.655804 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.654741    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\""
Oct 31 23:23:05.655804 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.654818    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 23:23:05.692394 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.689218    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-885fdf-nxpx8"]
Oct 31 23:23:05.757168 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.757125    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"]
Oct 31 23:23:05.757479 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.757449    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x" podUID="74a97819-25d1-43c8-b104-ee21999c5f68" containerName="istio-proxy" containerID="cri-o://b8a55f36c3f978f59a8e1afb3b32f9e273dd589f1308456c72eff537bbfac6c6" gracePeriod=30
Oct 31 23:23:05.806491 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.806457    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34d4f0d4-70aa-4c54-8a10-82677b2af2f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-885fdf-nxpx8\" (UID: \"34d4f0d4-70aa-4c54-8a10-82677b2af2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-885fdf-nxpx8"
Oct 31 23:23:05.806720 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.806530    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5ckd\" (UniqueName: \"kubernetes.io/projected/34d4f0d4-70aa-4c54-8a10-82677b2af2f2-kube-api-access-v5ckd\") pod \"maas-api-revoke-keys-e2e-api-b-885fdf-nxpx8\" (UID: \"34d4f0d4-70aa-4c54-8a10-82677b2af2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-885fdf-nxpx8"
Oct 31 23:23:05.806720 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.806689    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34d4f0d4-70aa-4c54-8a10-82677b2af2f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-885fdf-nxpx8\" (UID: \"34d4f0d4-70aa-4c54-8a10-82677b2af2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-885fdf-nxpx8"
Oct 31 23:23:05.933789 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.933653    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5ckd\" (UniqueName: \"kubernetes.io/projected/34d4f0d4-70aa-4c54-8a10-82677b2af2f2-kube-api-access-v5ckd\") pod \"maas-api-revoke-keys-e2e-api-b-885fdf-nxpx8\" (UID: \"34d4f0d4-70aa-4c54-8a10-82677b2af2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-885fdf-nxpx8"
Oct 31 23:23:05.933967 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.933825    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34d4f0d4-70aa-4c54-8a10-82677b2af2f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-885fdf-nxpx8\" (UID: \"34d4f0d4-70aa-4c54-8a10-82677b2af2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-885fdf-nxpx8"
Oct 31 23:23:05.934138 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.934118    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34d4f0d4-70aa-4c54-8a10-82677b2af2f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-885fdf-nxpx8\" (UID: \"34d4f0d4-70aa-4c54-8a10-82677b2af2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-885fdf-nxpx8"
Oct 31 23:23:05.935043 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.935018    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34d4f0d4-70aa-4c54-8a10-82677b2af2f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-885fdf-nxpx8\" (UID: \"34d4f0d4-70aa-4c54-8a10-82677b2af2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-885fdf-nxpx8"
Oct 31 23:23:05.993378 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:05.993322    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34d4f0d4-70aa-4c54-8a10-82677b2af2f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-885fdf-nxpx8\" (UID: \"34d4f0d4-70aa-4c54-8a10-82677b2af2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-885fdf-nxpx8"
Oct 31 23:23:06.033379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.030061    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5ckd\" (UniqueName: \"kubernetes.io/projected/34d4f0d4-70aa-4c54-8a10-82677b2af2f2-kube-api-access-v5ckd\") pod \"maas-api-revoke-keys-e2e-api-b-885fdf-nxpx8\" (UID: \"34d4f0d4-70aa-4c54-8a10-82677b2af2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-885fdf-nxpx8"
Oct 31 23:23:06.300037 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.298789    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-885fdf-nxpx8"
Oct 31 23:23:06.441701 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.441666    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba9088-nclfq"]
Oct 31 23:23:06.446123 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.446098    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba9088-nclfq"
Oct 31 23:23:06.471407 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.467740    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba9088-nclfq"]
Oct 31 23:23:06.569123 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.569049    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn7pv\" (UniqueName: \"kubernetes.io/projected/cdcd9409-f9de-4955-ab65-5a032181d22c-kube-api-access-kn7pv\") pod \"maas-api-revoke-keys-e2e-api-b-ba9088-nclfq\" (UID: \"cdcd9409-f9de-4955-ab65-5a032181d22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba9088-nclfq"
Oct 31 23:23:06.569477 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.569454    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdcd9409-f9de-4955-ab65-5a032181d22c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ba9088-nclfq\" (UID: \"cdcd9409-f9de-4955-ab65-5a032181d22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba9088-nclfq"
Oct 31 23:23:06.569604 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.569584    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdcd9409-f9de-4955-ab65-5a032181d22c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ba9088-nclfq\" (UID: \"cdcd9409-f9de-4955-ab65-5a032181d22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba9088-nclfq"
Oct 31 23:23:06.680452 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.677662    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn7pv\" (UniqueName: \"kubernetes.io/projected/cdcd9409-f9de-4955-ab65-5a032181d22c-kube-api-access-kn7pv\") pod \"maas-api-revoke-keys-e2e-api-b-ba9088-nclfq\" (UID: \"cdcd9409-f9de-4955-ab65-5a032181d22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba9088-nclfq"
Oct 31 23:23:06.680452 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.677783    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdcd9409-f9de-4955-ab65-5a032181d22c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ba9088-nclfq\" (UID: \"cdcd9409-f9de-4955-ab65-5a032181d22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba9088-nclfq"
Oct 31 23:23:06.680452 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.677885    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdcd9409-f9de-4955-ab65-5a032181d22c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ba9088-nclfq\" (UID: \"cdcd9409-f9de-4955-ab65-5a032181d22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba9088-nclfq"
Oct 31 23:23:06.680936 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.680695    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdcd9409-f9de-4955-ab65-5a032181d22c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ba9088-nclfq\" (UID: \"cdcd9409-f9de-4955-ab65-5a032181d22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba9088-nclfq"
Oct 31 23:23:06.683699 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.683669    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-885fdf-nxpx8"]
Oct 31 23:23:06.685045 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.684991    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdcd9409-f9de-4955-ab65-5a032181d22c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ba9088-nclfq\" (UID: \"cdcd9409-f9de-4955-ab65-5a032181d22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba9088-nclfq"
Oct 31 23:23:06.686433 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:23:06.686390    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d4f0d4_70aa_4c54_8a10_82677b2af2f2.slice/crio-8b6825953d675ac64df0363b571d92bdd67753485f61e0c3aca4182f9ca5abe7 WatchSource:0}: Error finding container 8b6825953d675ac64df0363b571d92bdd67753485f61e0c3aca4182f9ca5abe7: Status 404 returned error can't find the container with id 8b6825953d675ac64df0363b571d92bdd67753485f61e0c3aca4182f9ca5abe7
Oct 31 23:23:06.690335 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.690310    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn7pv\" (UniqueName: \"kubernetes.io/projected/cdcd9409-f9de-4955-ab65-5a032181d22c-kube-api-access-kn7pv\") pod \"maas-api-revoke-keys-e2e-api-b-ba9088-nclfq\" (UID: \"cdcd9409-f9de-4955-ab65-5a032181d22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba9088-nclfq"
Oct 31 23:23:06.766778 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.766752    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba9088-nclfq"
Oct 31 23:23:06.850545 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.850508    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-885fdf-nxpx8" event={"ID":"34d4f0d4-70aa-4c54-8a10-82677b2af2f2","Type":"ContainerStarted","Data":"8b6825953d675ac64df0363b571d92bdd67753485f61e0c3aca4182f9ca5abe7"}
Oct 31 23:23:06.869009 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.868979    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"]
Oct 31 23:23:06.875516 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.875492    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:23:06.881034 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.881000    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-istio-envoy\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:23:06.881179 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.881117    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34836e36-7a9c-4072-a555-970e2eb17c2b-istio-token\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:23:06.881179 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.881145    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-credential-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:23:06.881310 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.881267    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2t44\" (UniqueName: \"kubernetes.io/projected/34836e36-7a9c-4072-a555-970e2eb17c2b-kube-api-access-x2t44\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:23:06.881310 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.881299    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-istio-data\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:23:06.881450 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.881418    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34836e36-7a9c-4072-a555-970e2eb17c2b-istio-podinfo\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:23:06.881521 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.881445    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34836e36-7a9c-4072-a555-970e2eb17c2b-istiod-ca-cert\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:23:06.881521 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.881500    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-workload-certs\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:23:06.881646 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.881535    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-workload-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:23:06.891838 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.888482    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a1deb6-openshift-default-dockercfg-sx6st\""
Oct 31 23:23:06.896320 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.896298    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"]
Oct 31 23:23:06.982314 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.982281    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34836e36-7a9c-4072-a555-970e2eb17c2b-istio-token\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:23:06.982314 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.982315    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-credential-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:23:06.982563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.982354    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2t44\" (UniqueName: \"kubernetes.io/projected/34836e36-7a9c-4072-a555-970e2eb17c2b-kube-api-access-x2t44\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:23:06.982563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.982412    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-istio-data\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:23:06.982563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.982489    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34836e36-7a9c-4072-a555-970e2eb17c2b-istio-podinfo\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:23:06.982563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.982512    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34836e36-7a9c-4072-a555-970e2eb17c2b-istiod-ca-cert\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:23:06.982771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.982564    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-workload-certs\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:23:06.982771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.982599    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-workload-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:23:06.982771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.982690    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-istio-envoy\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:23:06.982935 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.982774    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-credential-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:23:06.982935 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.982841    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-istio-data\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:23:06.982935 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.982912    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-workload-certs\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:23:06.983109 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.983055    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-workload-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:23:06.983295 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.983274    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34836e36-7a9c-4072-a555-970e2eb17c2b-istiod-ca-cert\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:23:06.984954 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.984934    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-istio-envoy\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:23:06.985077 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:06.985058    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34836e36-7a9c-4072-a555-970e2eb17c2b-istio-podinfo\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:23:07.013259 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:07.013228    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34836e36-7a9c-4072-a555-970e2eb17c2b-istio-token\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:23:07.014861 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:07.014836    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2t44\" (UniqueName: \"kubernetes.io/projected/34836e36-7a9c-4072-a555-970e2eb17c2b-kube-api-access-x2t44\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:23:07.113546 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:07.113513    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba9088-nclfq"]
Oct 31 23:23:07.114236 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:23:07.114209    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdcd9409_f9de_4955_ab65_5a032181d22c.slice/crio-d501ed966f69e47f6e66b13a21882736b6b65cda705a47ed3a8e3de30f28a7bd WatchSource:0}: Error finding container d501ed966f69e47f6e66b13a21882736b6b65cda705a47ed3a8e3de30f28a7bd: Status 404 returned error can't find the container with id d501ed966f69e47f6e66b13a21882736b6b65cda705a47ed3a8e3de30f28a7bd
Oct 31 23:23:07.201114 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:07.201086    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:23:07.502300 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:07.502273    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"]
Oct 31 23:23:07.531028 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:23:07.530995    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34836e36_7a9c_4072_a555_970e2eb17c2b.slice/crio-ddd0608a9b54036977598ddb2b2e7df67e39e8065a1608c97e22d69bab076a6d WatchSource:0}: Error finding container ddd0608a9b54036977598ddb2b2e7df67e39e8065a1608c97e22d69bab076a6d: Status 404 returned error can't find the container with id ddd0608a9b54036977598ddb2b2e7df67e39e8065a1608c97e22d69bab076a6d
Oct 31 23:23:07.534788 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:07.534742    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:23:07.534888 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:07.534846    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:23:07.534927 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:07.534901    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:23:07.860055 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:07.860017    4301 generic.go:356] "Generic (PLEG): container finished" podID="34d4f0d4-70aa-4c54-8a10-82677b2af2f2" containerID="b0f29c1197dd0cf4d77eaaf49248ade2e2dc29fb32b2b103daa885a3b8b527b2" exitCode=0
Oct 31 23:23:07.860250 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:07.860102    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-885fdf-nxpx8" event={"ID":"34d4f0d4-70aa-4c54-8a10-82677b2af2f2","Type":"ContainerDied","Data":"b0f29c1197dd0cf4d77eaaf49248ade2e2dc29fb32b2b103daa885a3b8b527b2"}
Oct 31 23:23:07.861882 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:07.861855    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba9088-nclfq" event={"ID":"cdcd9409-f9de-4955-ab65-5a032181d22c","Type":"ContainerStarted","Data":"1f7e3b473785cb1acd986fdb37437e78cffb5afca5ab705abf121c842fcee035"}
Oct 31 23:23:07.862012 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:07.861899    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba9088-nclfq" event={"ID":"cdcd9409-f9de-4955-ab65-5a032181d22c","Type":"ContainerStarted","Data":"d501ed966f69e47f6e66b13a21882736b6b65cda705a47ed3a8e3de30f28a7bd"}
Oct 31 23:23:07.863579 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:07.863556    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd" event={"ID":"34836e36-7a9c-4072-a555-970e2eb17c2b","Type":"ContainerStarted","Data":"be25f67eed079add94747dc1a101d51b53b0d7fd3400cd273204bc930b358323"}
Oct 31 23:23:07.863693 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:07.863593    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd" event={"ID":"34836e36-7a9c-4072-a555-970e2eb17c2b","Type":"ContainerStarted","Data":"ddd0608a9b54036977598ddb2b2e7df67e39e8065a1608c97e22d69bab076a6d"}
Oct 31 23:23:07.920659 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:07.920445    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd" podStartSLOduration=1.920421906 podStartE2EDuration="1.920421906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:23:07.913944448 +0000 UTC m=+2006.661133957" watchObservedRunningTime="2026-10-31 23:23:07.920421906 +0000 UTC m=+2006.667611412"
Oct 31 23:23:07.975836 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:07.975766    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba9088-nclfq" podStartSLOduration=1.975743824 podStartE2EDuration="1.975743824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:23:07.946924511 +0000 UTC m=+2006.694114046" watchObservedRunningTime="2026-10-31 23:23:07.975743824 +0000 UTC m=+2006.722933330"
Oct 31 23:23:08.201816 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:08.201739    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:23:08.203251 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:08.203226    4301 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 31 23:23:08.203388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:08.203273    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd" podUID="34836e36-7a9c-4072-a555-970e2eb17c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 31 23:23:08.874658 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:08.874628    4301 generic.go:356] "Generic (PLEG): container finished" podID="cdcd9409-f9de-4955-ab65-5a032181d22c" containerID="1f7e3b473785cb1acd986fdb37437e78cffb5afca5ab705abf121c842fcee035" exitCode=0
Oct 31 23:23:08.875090 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:08.874721    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba9088-nclfq" event={"ID":"cdcd9409-f9de-4955-ab65-5a032181d22c","Type":"ContainerDied","Data":"1f7e3b473785cb1acd986fdb37437e78cffb5afca5ab705abf121c842fcee035"}
Oct 31 23:23:08.895836 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:08.895802    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 23:23:08.896021 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:08.895856    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 23:23:09.016839 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:09.016761    4301 patch_prober.go:29] interesting pod/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 23:23:09.017109 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:09.017075    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x" podUID="74a97819-25d1-43c8-b104-ee21999c5f68" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 23:23:09.091685 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:09.091661    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-885fdf-nxpx8"
Oct 31 23:23:09.161418 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:09.161303    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34d4f0d4-70aa-4c54-8a10-82677b2af2f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34d4f0d4-70aa-4c54-8a10-82677b2af2f2-maas-api-xks-ca\") pod \"34d4f0d4-70aa-4c54-8a10-82677b2af2f2\" (UID: \"34d4f0d4-70aa-4c54-8a10-82677b2af2f2\") "
Oct 31 23:23:09.161418 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:09.161416    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34d4f0d4-70aa-4c54-8a10-82677b2af2f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34d4f0d4-70aa-4c54-8a10-82677b2af2f2-maas-api-openshift-ca\") pod \"34d4f0d4-70aa-4c54-8a10-82677b2af2f2\" (UID: \"34d4f0d4-70aa-4c54-8a10-82677b2af2f2\") "
Oct 31 23:23:09.161670 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:09.161552    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34d4f0d4-70aa-4c54-8a10-82677b2af2f2-kube-api-access-v5ckd\" (UniqueName: \"kubernetes.io/projected/34d4f0d4-70aa-4c54-8a10-82677b2af2f2-kube-api-access-v5ckd\") pod \"34d4f0d4-70aa-4c54-8a10-82677b2af2f2\" (UID: \"34d4f0d4-70aa-4c54-8a10-82677b2af2f2\") "
Oct 31 23:23:09.162164 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:09.162132    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34d4f0d4-70aa-4c54-8a10-82677b2af2f2-maas-api-openshift-ca" pod "34d4f0d4-70aa-4c54-8a10-82677b2af2f2" (UID: "34d4f0d4-70aa-4c54-8a10-82677b2af2f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:23:09.162466 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:09.162439    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34d4f0d4-70aa-4c54-8a10-82677b2af2f2-maas-api-openshift-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:09.164074 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:09.164051    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34d4f0d4-70aa-4c54-8a10-82677b2af2f2-maas-api-xks-ca" pod "34d4f0d4-70aa-4c54-8a10-82677b2af2f2" (UID: "34d4f0d4-70aa-4c54-8a10-82677b2af2f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:23:09.164162 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:09.164125    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34d4f0d4-70aa-4c54-8a10-82677b2af2f2-kube-api-access-v5ckd" pod "34d4f0d4-70aa-4c54-8a10-82677b2af2f2" (UID: "34d4f0d4-70aa-4c54-8a10-82677b2af2f2"). InnerVolumeSpecName "kube-api-access-v5ckd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:23:09.202438 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:09.202407    4301 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 31 23:23:09.202577 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:09.202455    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd" podUID="34836e36-7a9c-4072-a555-970e2eb17c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 31 23:23:09.265381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:09.264157    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34d4f0d4-70aa-4c54-8a10-82677b2af2f2-maas-api-xks-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:09.265381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:09.264207    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5ckd\" (UniqueName: \"kubernetes.io/projected/34d4f0d4-70aa-4c54-8a10-82677b2af2f2-kube-api-access-v5ckd\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:09.275947 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:09.275915    4301 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 23:23:09.276114 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:09.275994    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99" podUID="4a05c486-5fda-45d2-95f7-57a8779a34a6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 23:23:09.883735 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:09.883693    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-885fdf-nxpx8"
Oct 31 23:23:09.884235 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:09.883739    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-885fdf-nxpx8" event={"ID":"34d4f0d4-70aa-4c54-8a10-82677b2af2f2","Type":"ContainerDied","Data":"8b6825953d675ac64df0363b571d92bdd67753485f61e0c3aca4182f9ca5abe7"}
Oct 31 23:23:09.884235 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:09.883782    4301 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b6825953d675ac64df0363b571d92bdd67753485f61e0c3aca4182f9ca5abe7"
Oct 31 23:23:10.201919 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.201892    4301 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 31 23:23:10.202055 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.201956    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd" podUID="34836e36-7a9c-4072-a555-970e2eb17c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 31 23:23:10.223825 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.223799    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba9088-nclfq"
Oct 31 23:23:10.387275 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.387240    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdcd9409-f9de-4955-ab65-5a032181d22c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdcd9409-f9de-4955-ab65-5a032181d22c-maas-api-xks-ca\") pod \"cdcd9409-f9de-4955-ab65-5a032181d22c\" (UID: \"cdcd9409-f9de-4955-ab65-5a032181d22c\") "
Oct 31 23:23:10.387275 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.387275    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdcd9409-f9de-4955-ab65-5a032181d22c-kube-api-access-kn7pv\" (UniqueName: \"kubernetes.io/projected/cdcd9409-f9de-4955-ab65-5a032181d22c-kube-api-access-kn7pv\") pod \"cdcd9409-f9de-4955-ab65-5a032181d22c\" (UID: \"cdcd9409-f9de-4955-ab65-5a032181d22c\") "
Oct 31 23:23:10.387635 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.387528    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdcd9409-f9de-4955-ab65-5a032181d22c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdcd9409-f9de-4955-ab65-5a032181d22c-maas-api-openshift-ca\") pod \"cdcd9409-f9de-4955-ab65-5a032181d22c\" (UID: \"cdcd9409-f9de-4955-ab65-5a032181d22c\") "
Oct 31 23:23:10.387919 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.387891    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdcd9409-f9de-4955-ab65-5a032181d22c-maas-api-openshift-ca" pod "cdcd9409-f9de-4955-ab65-5a032181d22c" (UID: "cdcd9409-f9de-4955-ab65-5a032181d22c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:23:10.388237 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.388183    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdcd9409-f9de-4955-ab65-5a032181d22c-maas-api-openshift-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:10.389709 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.389687    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdcd9409-f9de-4955-ab65-5a032181d22c-kube-api-access-kn7pv" pod "cdcd9409-f9de-4955-ab65-5a032181d22c" (UID: "cdcd9409-f9de-4955-ab65-5a032181d22c"). InnerVolumeSpecName "kube-api-access-kn7pv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:23:10.389936 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.389921    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdcd9409-f9de-4955-ab65-5a032181d22c-maas-api-xks-ca" pod "cdcd9409-f9de-4955-ab65-5a032181d22c" (UID: "cdcd9409-f9de-4955-ab65-5a032181d22c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:23:10.424313 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.424285    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:23:10.489947 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.489912    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdcd9409-f9de-4955-ab65-5a032181d22c-maas-api-xks-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:10.489947 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.489948    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn7pv\" (UniqueName: \"kubernetes.io/projected/cdcd9409-f9de-4955-ab65-5a032181d22c-kube-api-access-kn7pv\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:10.608928 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.593937    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-istio-data\") pod \"4a05c486-5fda-45d2-95f7-57a8779a34a6\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") "
Oct 31 23:23:10.608928 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.594005    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-workload-socket\") pod \"4a05c486-5fda-45d2-95f7-57a8779a34a6\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") "
Oct 31 23:23:10.608928 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.594055    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a05c486-5fda-45d2-95f7-57a8779a34a6-istio-token\" (UniqueName: \"kubernetes.io/projected/4a05c486-5fda-45d2-95f7-57a8779a34a6-istio-token\") pod \"4a05c486-5fda-45d2-95f7-57a8779a34a6\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") "
Oct 31 23:23:10.608928 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.594159    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-credential-socket\") pod \"4a05c486-5fda-45d2-95f7-57a8779a34a6\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") "
Oct 31 23:23:10.608928 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.594203    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a05c486-5fda-45d2-95f7-57a8779a34a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a05c486-5fda-45d2-95f7-57a8779a34a6-istio-podinfo\") pod \"4a05c486-5fda-45d2-95f7-57a8779a34a6\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") "
Oct 31 23:23:10.608928 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.594311    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-istio-envoy\") pod \"4a05c486-5fda-45d2-95f7-57a8779a34a6\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") "
Oct 31 23:23:10.608928 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.594384    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a05c486-5fda-45d2-95f7-57a8779a34a6-kube-api-access-ntd6n\" (UniqueName: \"kubernetes.io/projected/4a05c486-5fda-45d2-95f7-57a8779a34a6-kube-api-access-ntd6n\") pod \"4a05c486-5fda-45d2-95f7-57a8779a34a6\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") "
Oct 31 23:23:10.608928 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.594514    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a05c486-5fda-45d2-95f7-57a8779a34a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a05c486-5fda-45d2-95f7-57a8779a34a6-istiod-ca-cert\") pod \"4a05c486-5fda-45d2-95f7-57a8779a34a6\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") "
Oct 31 23:23:10.608928 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.594581    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-workload-certs\") pod \"4a05c486-5fda-45d2-95f7-57a8779a34a6\" (UID: \"4a05c486-5fda-45d2-95f7-57a8779a34a6\") "
Oct 31 23:23:10.608928 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.607854    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-workload-certs" pod "4a05c486-5fda-45d2-95f7-57a8779a34a6" (UID: "4a05c486-5fda-45d2-95f7-57a8779a34a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:10.608928 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.608281    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-credential-socket" pod "4a05c486-5fda-45d2-95f7-57a8779a34a6" (UID: "4a05c486-5fda-45d2-95f7-57a8779a34a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:10.608928 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.608580    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-istio-data" pod "4a05c486-5fda-45d2-95f7-57a8779a34a6" (UID: "4a05c486-5fda-45d2-95f7-57a8779a34a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:10.608928 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.608811    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-workload-socket" pod "4a05c486-5fda-45d2-95f7-57a8779a34a6" (UID: "4a05c486-5fda-45d2-95f7-57a8779a34a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:10.610058 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.609721    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a05c486-5fda-45d2-95f7-57a8779a34a6-istiod-ca-cert" pod "4a05c486-5fda-45d2-95f7-57a8779a34a6" (UID: "4a05c486-5fda-45d2-95f7-57a8779a34a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:23:10.674745 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.672384    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a05c486-5fda-45d2-95f7-57a8779a34a6-kube-api-access-ntd6n" pod "4a05c486-5fda-45d2-95f7-57a8779a34a6" (UID: "4a05c486-5fda-45d2-95f7-57a8779a34a6"). InnerVolumeSpecName "kube-api-access-ntd6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:23:10.674745 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.672509    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a05c486-5fda-45d2-95f7-57a8779a34a6-istio-token" pod "4a05c486-5fda-45d2-95f7-57a8779a34a6" (UID: "4a05c486-5fda-45d2-95f7-57a8779a34a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:23:10.674745 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.673021    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-istio-envoy" pod "4a05c486-5fda-45d2-95f7-57a8779a34a6" (UID: "4a05c486-5fda-45d2-95f7-57a8779a34a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:10.674745 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.674627    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a05c486-5fda-45d2-95f7-57a8779a34a6-istio-podinfo" pod "4a05c486-5fda-45d2-95f7-57a8779a34a6" (UID: "4a05c486-5fda-45d2-95f7-57a8779a34a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:23:10.721581 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.721410    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a05c486-5fda-45d2-95f7-57a8779a34a6-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:10.721581 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.721445    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:10.721581 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.721458    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:10.721581 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.721471    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:10.721581 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.721485    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a05c486-5fda-45d2-95f7-57a8779a34a6-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:10.721581 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.721499    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:10.721581 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.721512    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a05c486-5fda-45d2-95f7-57a8779a34a6-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:10.721581 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.721523    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a05c486-5fda-45d2-95f7-57a8779a34a6-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:10.721581 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.721535    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntd6n\" (UniqueName: \"kubernetes.io/projected/4a05c486-5fda-45d2-95f7-57a8779a34a6-kube-api-access-ntd6n\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:10.898263 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.897914    4301 generic.go:356] "Generic (PLEG): container finished" podID="4a05c486-5fda-45d2-95f7-57a8779a34a6" containerID="693875dcdf09637209b08e5a73905fd483501e815d9549fb15b01df72fc7a381" exitCode=0
Oct 31 23:23:10.898263 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.898029    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99" event={"ID":"4a05c486-5fda-45d2-95f7-57a8779a34a6","Type":"ContainerDied","Data":"693875dcdf09637209b08e5a73905fd483501e815d9549fb15b01df72fc7a381"}
Oct 31 23:23:10.898263 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.898067    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99" event={"ID":"4a05c486-5fda-45d2-95f7-57a8779a34a6","Type":"ContainerDied","Data":"42236602974be242d4aa5a317d032d6cf74d0da5484066510cf5c24803dad643"}
Oct 31 23:23:10.898263 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.898089    4301 scope.go:122] "RemoveContainer" containerID="693875dcdf09637209b08e5a73905fd483501e815d9549fb15b01df72fc7a381"
Oct 31 23:23:10.898263 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.898202    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"
Oct 31 23:23:10.901421 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.900295    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:23:10.904146 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.904123    4301 generic.go:356] "Generic (PLEG): container finished" podID="f7560a60-cbba-43ba-81a1-946c57663b80" containerID="6b48c06c7ecfdd2a87f194966f50cd3ba912dcfd514ea4cf071f07ce8a6ef305" exitCode=0
Oct 31 23:23:10.904239 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.904192    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" event={"ID":"f7560a60-cbba-43ba-81a1-946c57663b80","Type":"ContainerDied","Data":"6b48c06c7ecfdd2a87f194966f50cd3ba912dcfd514ea4cf071f07ce8a6ef305"}
Oct 31 23:23:10.904280 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.904251    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq" event={"ID":"f7560a60-cbba-43ba-81a1-946c57663b80","Type":"ContainerDied","Data":"37dfcc5efb26c796f7a89abb1d5a6e151608f42af2988151a6476650531583e5"}
Oct 31 23:23:10.907823 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.907798    4301 generic.go:356] "Generic (PLEG): container finished" podID="74a97819-25d1-43c8-b104-ee21999c5f68" containerID="b8a55f36c3f978f59a8e1afb3b32f9e273dd589f1308456c72eff537bbfac6c6" exitCode=0
Oct 31 23:23:10.908228 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.908207    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x" event={"ID":"74a97819-25d1-43c8-b104-ee21999c5f68","Type":"ContainerDied","Data":"b8a55f36c3f978f59a8e1afb3b32f9e273dd589f1308456c72eff537bbfac6c6"}
Oct 31 23:23:10.915261 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.915230    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba9088-nclfq" event={"ID":"cdcd9409-f9de-4955-ab65-5a032181d22c","Type":"ContainerDied","Data":"d501ed966f69e47f6e66b13a21882736b6b65cda705a47ed3a8e3de30f28a7bd"}
Oct 31 23:23:10.915721 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.915705    4301 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d501ed966f69e47f6e66b13a21882736b6b65cda705a47ed3a8e3de30f28a7bd"
Oct 31 23:23:10.915838 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.915596    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba9088-nclfq"
Oct 31 23:23:10.918113 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.918100    4301 scope.go:122] "RemoveContainer" containerID="693875dcdf09637209b08e5a73905fd483501e815d9549fb15b01df72fc7a381"
Oct 31 23:23:10.918666 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:23:10.918532    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"693875dcdf09637209b08e5a73905fd483501e815d9549fb15b01df72fc7a381\": container with ID starting with 693875dcdf09637209b08e5a73905fd483501e815d9549fb15b01df72fc7a381 not found: ID does not exist" containerID="693875dcdf09637209b08e5a73905fd483501e815d9549fb15b01df72fc7a381"
Oct 31 23:23:10.918666 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.918566    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"693875dcdf09637209b08e5a73905fd483501e815d9549fb15b01df72fc7a381"} err="failed to get container status \"693875dcdf09637209b08e5a73905fd483501e815d9549fb15b01df72fc7a381\": rpc error: code = NotFound desc = could not find container \"693875dcdf09637209b08e5a73905fd483501e815d9549fb15b01df72fc7a381\": container with ID starting with 693875dcdf09637209b08e5a73905fd483501e815d9549fb15b01df72fc7a381 not found: ID does not exist"
Oct 31 23:23:10.918666 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.918589    4301 scope.go:122] "RemoveContainer" containerID="6b48c06c7ecfdd2a87f194966f50cd3ba912dcfd514ea4cf071f07ce8a6ef305"
Oct 31 23:23:10.926645 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.923811    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-workload-certs\") pod \"f7560a60-cbba-43ba-81a1-946c57663b80\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") "
Oct 31 23:23:10.926645 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.923911    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7560a60-cbba-43ba-81a1-946c57663b80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7560a60-cbba-43ba-81a1-946c57663b80-istio-podinfo\") pod \"f7560a60-cbba-43ba-81a1-946c57663b80\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") "
Oct 31 23:23:10.926645 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.923945    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-workload-socket\") pod \"f7560a60-cbba-43ba-81a1-946c57663b80\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") "
Oct 31 23:23:10.926645 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.924002    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-istio-envoy\") pod \"f7560a60-cbba-43ba-81a1-946c57663b80\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") "
Oct 31 23:23:10.926645 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.924154    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-istio-data\") pod \"f7560a60-cbba-43ba-81a1-946c57663b80\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") "
Oct 31 23:23:10.926645 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.924287    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-credential-socket\") pod \"f7560a60-cbba-43ba-81a1-946c57663b80\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") "
Oct 31 23:23:10.926645 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.924376    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7560a60-cbba-43ba-81a1-946c57663b80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7560a60-cbba-43ba-81a1-946c57663b80-istiod-ca-cert\") pod \"f7560a60-cbba-43ba-81a1-946c57663b80\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") "
Oct 31 23:23:10.926645 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.924499    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7560a60-cbba-43ba-81a1-946c57663b80-istio-token\" (UniqueName: \"kubernetes.io/projected/f7560a60-cbba-43ba-81a1-946c57663b80-istio-token\") pod \"f7560a60-cbba-43ba-81a1-946c57663b80\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") "
Oct 31 23:23:10.926645 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.924534    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7560a60-cbba-43ba-81a1-946c57663b80-kube-api-access-8hshl\" (UniqueName: \"kubernetes.io/projected/f7560a60-cbba-43ba-81a1-946c57663b80-kube-api-access-8hshl\") pod \"f7560a60-cbba-43ba-81a1-946c57663b80\" (UID: \"f7560a60-cbba-43ba-81a1-946c57663b80\") "
Oct 31 23:23:10.961851 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.961452    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-workload-certs" pod "f7560a60-cbba-43ba-81a1-946c57663b80" (UID: "f7560a60-cbba-43ba-81a1-946c57663b80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:10.965023 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.964828    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7560a60-cbba-43ba-81a1-946c57663b80-istio-podinfo" pod "f7560a60-cbba-43ba-81a1-946c57663b80" (UID: "f7560a60-cbba-43ba-81a1-946c57663b80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:23:10.965584 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.965285    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-workload-socket" pod "f7560a60-cbba-43ba-81a1-946c57663b80" (UID: "f7560a60-cbba-43ba-81a1-946c57663b80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:10.966865 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.966789    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7560a60-cbba-43ba-81a1-946c57663b80-istiod-ca-cert" pod "f7560a60-cbba-43ba-81a1-946c57663b80" (UID: "f7560a60-cbba-43ba-81a1-946c57663b80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:23:10.967294 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.967255    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-credential-socket" pod "f7560a60-cbba-43ba-81a1-946c57663b80" (UID: "f7560a60-cbba-43ba-81a1-946c57663b80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:10.967944 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.967598    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-istio-data" pod "f7560a60-cbba-43ba-81a1-946c57663b80" (UID: "f7560a60-cbba-43ba-81a1-946c57663b80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:10.970088 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.970043    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7560a60-cbba-43ba-81a1-946c57663b80-kube-api-access-8hshl" pod "f7560a60-cbba-43ba-81a1-946c57663b80" (UID: "f7560a60-cbba-43ba-81a1-946c57663b80"). InnerVolumeSpecName "kube-api-access-8hshl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:23:10.970781 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.970760    4301 scope.go:122] "RemoveContainer" containerID="aa732f12d6a9ce2ff5978da02c04bf10f8256e93d9e297a2cbd2d997864ada20"
Oct 31 23:23:10.974462 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.974398    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-istio-envoy" pod "f7560a60-cbba-43ba-81a1-946c57663b80" (UID: "f7560a60-cbba-43ba-81a1-946c57663b80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:10.982058 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.982016    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7560a60-cbba-43ba-81a1-946c57663b80-istio-token" pod "f7560a60-cbba-43ba-81a1-946c57663b80" (UID: "f7560a60-cbba-43ba-81a1-946c57663b80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:23:10.984750 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.984700    4301 scope.go:122] "RemoveContainer" containerID="6b48c06c7ecfdd2a87f194966f50cd3ba912dcfd514ea4cf071f07ce8a6ef305"
Oct 31 23:23:10.985131 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:23:10.985014    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b48c06c7ecfdd2a87f194966f50cd3ba912dcfd514ea4cf071f07ce8a6ef305\": container with ID starting with 6b48c06c7ecfdd2a87f194966f50cd3ba912dcfd514ea4cf071f07ce8a6ef305 not found: ID does not exist" containerID="6b48c06c7ecfdd2a87f194966f50cd3ba912dcfd514ea4cf071f07ce8a6ef305"
Oct 31 23:23:10.985131 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.985042    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b48c06c7ecfdd2a87f194966f50cd3ba912dcfd514ea4cf071f07ce8a6ef305"} err="failed to get container status \"6b48c06c7ecfdd2a87f194966f50cd3ba912dcfd514ea4cf071f07ce8a6ef305\": rpc error: code = NotFound desc = could not find container \"6b48c06c7ecfdd2a87f194966f50cd3ba912dcfd514ea4cf071f07ce8a6ef305\": container with ID starting with 6b48c06c7ecfdd2a87f194966f50cd3ba912dcfd514ea4cf071f07ce8a6ef305 not found: ID does not exist"
Oct 31 23:23:10.985131 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.985060    4301 scope.go:122] "RemoveContainer" containerID="aa732f12d6a9ce2ff5978da02c04bf10f8256e93d9e297a2cbd2d997864ada20"
Oct 31 23:23:10.985552 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:23:10.985436    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa732f12d6a9ce2ff5978da02c04bf10f8256e93d9e297a2cbd2d997864ada20\": container with ID starting with aa732f12d6a9ce2ff5978da02c04bf10f8256e93d9e297a2cbd2d997864ada20 not found: ID does not exist" containerID="aa732f12d6a9ce2ff5978da02c04bf10f8256e93d9e297a2cbd2d997864ada20"
Oct 31 23:23:10.985552 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.985461    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa732f12d6a9ce2ff5978da02c04bf10f8256e93d9e297a2cbd2d997864ada20"} err="failed to get container status \"aa732f12d6a9ce2ff5978da02c04bf10f8256e93d9e297a2cbd2d997864ada20\": rpc error: code = NotFound desc = could not find container \"aa732f12d6a9ce2ff5978da02c04bf10f8256e93d9e297a2cbd2d997864ada20\": container with ID starting with aa732f12d6a9ce2ff5978da02c04bf10f8256e93d9e297a2cbd2d997864ada20 not found: ID does not exist"
Oct 31 23:23:10.985552 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.985479    4301 scope.go:122] "RemoveContainer" containerID="9457e388751789f92d5088d58f4cd2dda892459c3ddd36288972da7dea82957f"
Oct 31 23:23:10.994113 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.994092    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"]
Oct 31 23:23:10.999650 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:10.999589    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-szd99"]
Oct 31 23:23:11.027175 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.026950    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7560a60-cbba-43ba-81a1-946c57663b80-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:11.027175 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.026984    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hshl\" (UniqueName: \"kubernetes.io/projected/f7560a60-cbba-43ba-81a1-946c57663b80-kube-api-access-8hshl\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:11.027175 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.027002    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:11.027175 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.027017    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7560a60-cbba-43ba-81a1-946c57663b80-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:11.027175 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.027031    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:11.027175 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.027045    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:11.027175 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.027058    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:11.027175 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.027072    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7560a60-cbba-43ba-81a1-946c57663b80-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:11.027175 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.027085    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7560a60-cbba-43ba-81a1-946c57663b80-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:11.208570 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.201856    4301 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 31 23:23:11.208570 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.201916    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd" podUID="34836e36-7a9c-4072-a555-970e2eb17c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 31 23:23:11.415492 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.415464    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a05c486-5fda-45d2-95f7-57a8779a34a6" path="/var/lib/kubelet/pods/4a05c486-5fda-45d2-95f7-57a8779a34a6/volumes"
Oct 31 23:23:11.416670 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.416652    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:23:11.562447 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.560484    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74a97819-25d1-43c8-b104-ee21999c5f68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a97819-25d1-43c8-b104-ee21999c5f68-istiod-ca-cert\") pod \"74a97819-25d1-43c8-b104-ee21999c5f68\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") "
Oct 31 23:23:11.562447 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.560723    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a97819-25d1-43c8-b104-ee21999c5f68-istio-token\" (UniqueName: \"kubernetes.io/projected/74a97819-25d1-43c8-b104-ee21999c5f68-istio-token\") pod \"74a97819-25d1-43c8-b104-ee21999c5f68\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") "
Oct 31 23:23:11.562447 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.560783    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-workload-certs\") pod \"74a97819-25d1-43c8-b104-ee21999c5f68\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") "
Oct 31 23:23:11.562447 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.560813    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a97819-25d1-43c8-b104-ee21999c5f68-kube-api-access-459hz\" (UniqueName: \"kubernetes.io/projected/74a97819-25d1-43c8-b104-ee21999c5f68-kube-api-access-459hz\") pod \"74a97819-25d1-43c8-b104-ee21999c5f68\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") "
Oct 31 23:23:11.562447 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.560991    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-workload-socket\") pod \"74a97819-25d1-43c8-b104-ee21999c5f68\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") "
Oct 31 23:23:11.562447 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.561155    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-credential-socket\") pod \"74a97819-25d1-43c8-b104-ee21999c5f68\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") "
Oct 31 23:23:11.562447 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.561186    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-istio-data\") pod \"74a97819-25d1-43c8-b104-ee21999c5f68\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") "
Oct 31 23:23:11.562447 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.561240    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-istio-envoy\") pod \"74a97819-25d1-43c8-b104-ee21999c5f68\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") "
Oct 31 23:23:11.597723 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.590564    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-workload-certs" pod "74a97819-25d1-43c8-b104-ee21999c5f68" (UID: "74a97819-25d1-43c8-b104-ee21999c5f68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:11.606904 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.606435    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a97819-25d1-43c8-b104-ee21999c5f68-istiod-ca-cert" pod "74a97819-25d1-43c8-b104-ee21999c5f68" (UID: "74a97819-25d1-43c8-b104-ee21999c5f68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:23:11.618407 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.609863    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-workload-socket" pod "74a97819-25d1-43c8-b104-ee21999c5f68" (UID: "74a97819-25d1-43c8-b104-ee21999c5f68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:11.618407 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.612476    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74a97819-25d1-43c8-b104-ee21999c5f68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a97819-25d1-43c8-b104-ee21999c5f68-istio-podinfo\") pod \"74a97819-25d1-43c8-b104-ee21999c5f68\" (UID: \"74a97819-25d1-43c8-b104-ee21999c5f68\") "
Oct 31 23:23:11.645527 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.628749    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74a97819-25d1-43c8-b104-ee21999c5f68-istio-podinfo" pod "74a97819-25d1-43c8-b104-ee21999c5f68" (UID: "74a97819-25d1-43c8-b104-ee21999c5f68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:23:11.645527 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.630414    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-istio-data" pod "74a97819-25d1-43c8-b104-ee21999c5f68" (UID: "74a97819-25d1-43c8-b104-ee21999c5f68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:11.645527 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.645469    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-credential-socket" pod "74a97819-25d1-43c8-b104-ee21999c5f68" (UID: "74a97819-25d1-43c8-b104-ee21999c5f68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:11.652558 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.651495    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"]
Oct 31 23:23:11.652762 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.652563    4301 cpu_manager.go:414] "containerMap: removing container" podUID="34d4f0d4-70aa-4c54-8a10-82677b2af2f2" containerName="revoke-keys"
Oct 31 23:23:11.652762 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.652591    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34d4f0d4-70aa-4c54-8a10-82677b2af2f2" containerName="revoke-keys"
Oct 31 23:23:11.652762 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.652607    4301 cpu_manager.go:414] "containerMap: removing container" podUID="74a97819-25d1-43c8-b104-ee21999c5f68" containerName="istio-proxy"
Oct 31 23:23:11.652762 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.652616    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74a97819-25d1-43c8-b104-ee21999c5f68" containerName="istio-proxy"
Oct 31 23:23:11.653909 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.653880    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:11.654185 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.654168    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:11.654284 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.654272    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:11.654417 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.654402    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:11.654532 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.654521    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a97819-25d1-43c8-b104-ee21999c5f68-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:11.654627 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.654616    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a97819-25d1-43c8-b104-ee21999c5f68-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:11.654699 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.653951    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a97819-25d1-43c8-b104-ee21999c5f68-istio-token" pod "74a97819-25d1-43c8-b104-ee21999c5f68" (UID: "74a97819-25d1-43c8-b104-ee21999c5f68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:23:11.656856 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.656810    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-istio-envoy" pod "74a97819-25d1-43c8-b104-ee21999c5f68" (UID: "74a97819-25d1-43c8-b104-ee21999c5f68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:11.659766 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.659729    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a97819-25d1-43c8-b104-ee21999c5f68-kube-api-access-459hz" pod "74a97819-25d1-43c8-b104-ee21999c5f68" (UID: "74a97819-25d1-43c8-b104-ee21999c5f68"). InnerVolumeSpecName "kube-api-access-459hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:23:11.664452 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.662447    4301 cpu_manager.go:414] "containerMap: removing container" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy"
Oct 31 23:23:11.664452 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.662473    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy"
Oct 31 23:23:11.664452 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.662494    4301 cpu_manager.go:414] "containerMap: removing container" podUID="cdcd9409-f9de-4955-ab65-5a032181d22c" containerName="revoke-keys"
Oct 31 23:23:11.664452 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.662502    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdcd9409-f9de-4955-ab65-5a032181d22c" containerName="revoke-keys"
Oct 31 23:23:11.664452 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.662529    4301 cpu_manager.go:414] "containerMap: removing container" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy"
Oct 31 23:23:11.664452 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.662536    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy"
Oct 31 23:23:11.664452 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.662550    4301 cpu_manager.go:414] "containerMap: removing container" podUID="4a05c486-5fda-45d2-95f7-57a8779a34a6" containerName="istio-proxy"
Oct 31 23:23:11.664452 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.662558    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a05c486-5fda-45d2-95f7-57a8779a34a6" containerName="istio-proxy"
Oct 31 23:23:11.689406 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.675531    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy"
Oct 31 23:23:11.689406 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.675567    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="34d4f0d4-70aa-4c54-8a10-82677b2af2f2" containerName="revoke-keys"
Oct 31 23:23:11.689406 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.675579    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="74a97819-25d1-43c8-b104-ee21999c5f68" containerName="istio-proxy"
Oct 31 23:23:11.689406 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.675593    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdcd9409-f9de-4955-ab65-5a032181d22c" containerName="revoke-keys"
Oct 31 23:23:11.689406 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.675603    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy"
Oct 31 23:23:11.689406 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.675612    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a05c486-5fda-45d2-95f7-57a8779a34a6" containerName="istio-proxy"
Oct 31 23:23:11.689406 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.675624    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="74a97819-25d1-43c8-b104-ee21999c5f68" containerName="istio-proxy"
Oct 31 23:23:11.693185 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.693157    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:23:11.693865 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.693813    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"]
Oct 31 23:23:11.699763 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.699624    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"]
Oct 31 23:23:11.700486 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.700463    4301 cpu_manager.go:414] "containerMap: removing container" podUID="74a97819-25d1-43c8-b104-ee21999c5f68" containerName="istio-proxy"
Oct 31 23:23:11.700486 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.700486    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74a97819-25d1-43c8-b104-ee21999c5f68" containerName="istio-proxy"
Oct 31 23:23:11.700635 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.700532    4301 cpu_manager.go:414] "containerMap: removing container" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy"
Oct 31 23:23:11.700635 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.700541    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy"
Oct 31 23:23:11.700760 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.700692    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" containerName="istio-proxy"
Oct 31 23:23:11.717980 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.717947    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"]
Oct 31 23:23:11.718130 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.718083    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:23:11.766387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.759303    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istio-envoy\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:23:11.766941 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.766908    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-istio-envoy\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:23:11.767184 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.767162    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba885b3-4cbf-4677-bef9-94f492ca8794-istiod-ca-cert\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:23:11.767350 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.767331    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-istio-data\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:23:11.767587 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.767560    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-workload-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:23:11.773003 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.772911    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba885b3-4cbf-4677-bef9-94f492ca8794-istio-podinfo\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:23:11.773141 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.773007    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-workload-certs\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:23:11.773141 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.773072    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istio-token\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:23:11.773141 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.773106    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba885b3-4cbf-4677-bef9-94f492ca8794-istio-token\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:23:11.773315 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.773145    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-credential-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:23:11.773315 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.773243    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8jrv\" (UniqueName: \"kubernetes.io/projected/bba885b3-4cbf-4677-bef9-94f492ca8794-kube-api-access-m8jrv\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:23:11.773315 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.773303    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-workload-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:23:11.773514 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.773337    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-workload-certs\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:23:11.774329 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.774288    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9tbw\" (UniqueName: \"kubernetes.io/projected/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-kube-api-access-t9tbw\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:23:11.774814 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.774782    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istio-data\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:23:11.774957 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.774939    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istio-podinfo\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:23:11.775106 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.775091    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-credential-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:23:11.775299 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.775272    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istiod-ca-cert\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:23:11.775667 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.775637    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a97819-25d1-43c8-b104-ee21999c5f68-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:11.775813 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.775797    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a97819-25d1-43c8-b104-ee21999c5f68-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:11.775908 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.775896    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-459hz\" (UniqueName: \"kubernetes.io/projected/74a97819-25d1-43c8-b104-ee21999c5f68-kube-api-access-459hz\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:11.894468 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.894423    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istio-envoy\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:23:11.896530 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.896502    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-istio-envoy\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:23:11.902506 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.902471    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba885b3-4cbf-4677-bef9-94f492ca8794-istiod-ca-cert\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:23:11.909729 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.903115    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-istio-data\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:23:11.921489 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.921450    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-workload-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:23:11.921767 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.921741    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba885b3-4cbf-4677-bef9-94f492ca8794-istio-podinfo\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:23:11.921949 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.921930    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-workload-certs\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:23:11.922110 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.922093    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istio-token\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:23:11.922250 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.922234    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba885b3-4cbf-4677-bef9-94f492ca8794-istio-token\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:23:11.922402 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.922386    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-credential-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:23:11.922652 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.922628    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8jrv\" (UniqueName: \"kubernetes.io/projected/bba885b3-4cbf-4677-bef9-94f492ca8794-kube-api-access-m8jrv\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:23:11.922834 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.922816    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-workload-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:23:11.922955 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.922940    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-workload-certs\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:23:11.941797 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.941767    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9tbw\" (UniqueName: \"kubernetes.io/projected/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-kube-api-access-t9tbw\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:23:11.942061 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.942039    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istio-data\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:23:11.942216 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.942201    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istio-podinfo\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:23:11.942379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.942342    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-credential-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:23:11.942634 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.942614    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istiod-ca-cert\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:23:11.946773 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.943596    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istiod-ca-cert\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:23:11.949212 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.947118    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-workload-certs\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:23:11.949212 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.915212    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba885b3-4cbf-4677-bef9-94f492ca8794-istiod-ca-cert\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:23:11.949212 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.915566    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-istio-data\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:23:11.949212 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.946378    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-credential-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:23:11.949212 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.933679    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-workload-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:23:11.949212 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.931033    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-workload-certs\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:23:11.963969 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.961756    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istio-podinfo\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:23:11.963969 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.962336    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8jrv\" (UniqueName: \"kubernetes.io/projected/bba885b3-4cbf-4677-bef9-94f492ca8794-kube-api-access-m8jrv\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:23:11.968446 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.944729    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-workload-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:23:11.968891 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.968861    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istio-data\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:23:11.969303 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.969282    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-credential-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:23:11.998380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.997085    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x" event={"ID":"74a97819-25d1-43c8-b104-ee21999c5f68","Type":"ContainerDied","Data":"ae0eb8e55f3e7779a6b2b20e4ef3bf3863a4f0911774f770a477fc8b20bb566f"}
Oct 31 23:23:11.998380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.997163    4301 scope.go:122] "RemoveContainer" containerID="b8a55f36c3f978f59a8e1afb3b32f9e273dd589f1308456c72eff537bbfac6c6"
Oct 31 23:23:11.998380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:11.997385    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"
Oct 31 23:23:12.025466 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.024294    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"]
Oct 31 23:23:12.062795 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.062764    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:12.082234 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.082196    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9tbw\" (UniqueName: \"kubernetes.io/projected/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-kube-api-access-t9tbw\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:23:12.105381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.090776    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istio-token\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:23:12.109687 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.109655    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"]
Oct 31 23:23:12.141018 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.117215    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba885b3-4cbf-4677-bef9-94f492ca8794-istio-podinfo\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:23:12.141422 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.117491    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"
Oct 31 23:23:12.142350 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.128340    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-istio-envoy\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:23:12.142883 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.128636    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istio-envoy\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:23:12.167427 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.146722    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba885b3-4cbf-4677-bef9-94f492ca8794-istio-token\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:23:12.204395 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.193328    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"]
Oct 31 23:23:12.226412 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.211182    4301 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 31 23:23:12.226412 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.211272    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd" podUID="34836e36-7a9c-4072-a555-970e2eb17c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 31 23:23:12.226412 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.212188    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"]
Oct 31 23:23:12.226412 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.212294    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:12.260394 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.259677    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istio-podinfo\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:12.260394 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.259735    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-workload-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:12.260394 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.259869    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istio-data\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:12.260394 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.259896    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-credential-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:12.260394 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.259921    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vgjv\" (UniqueName: \"kubernetes.io/projected/e5c2b8fb-331d-4df2-ad30-13549d8a660c-kube-api-access-2vgjv\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:12.260394 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.259962    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istiod-ca-cert\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:12.260394 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.259985    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-workload-certs\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:12.260394 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.260091    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istio-token\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:12.260394 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.260233    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istio-envoy\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:12.294721 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.294697    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"]
Oct 31 23:23:12.299775 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.299747    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-k8nzq"]
Oct 31 23:23:12.320549 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.320511    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"]
Oct 31 23:23:12.320852 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.320834    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:23:12.324654 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.324631    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-4528ed-openshift-default-b7757d6b5-9296x"]
Oct 31 23:23:12.363987 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.361692    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istio-envoy\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:12.363987 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.361797    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-istio-envoy\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:12.363987 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.361849    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istio-podinfo\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:12.363987 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.361884    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-workload-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:12.363987 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.361917    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-credential-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:12.363987 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.362047    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istio-data\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:12.363987 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.362076    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-credential-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:12.363987 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.362101    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vgjv\" (UniqueName: \"kubernetes.io/projected/e5c2b8fb-331d-4df2-ad30-13549d8a660c-kube-api-access-2vgjv\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:12.363987 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.362140    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1a1157d-88ea-448f-931d-9528c5df5684-istio-token\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:12.363987 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.362177    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istiod-ca-cert\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:12.363987 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.362202    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1a1157d-88ea-448f-931d-9528c5df5684-istio-podinfo\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:12.363987 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.362229    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-workload-certs\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:12.363987 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.362276    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-workload-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:12.363987 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.362308    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1a1157d-88ea-448f-931d-9528c5df5684-istiod-ca-cert\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:12.363987 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.362951    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-credential-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:12.368490 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.365111    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-workload-certs\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:12.368490 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.365165    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbxrn\" (UniqueName: \"kubernetes.io/projected/b1a1157d-88ea-448f-931d-9528c5df5684-kube-api-access-zbxrn\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:12.368490 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.365198    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istio-token\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:12.368490 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.365352    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-istio-data\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:12.368490 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.366660    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-workload-certs\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:12.368490 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.366904    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-workload-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:12.368490 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.367843    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istiod-ca-cert\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:12.368490 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.368398    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istio-data\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:12.369291 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.369232    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istio-envoy\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:12.371191 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.371142    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istio-podinfo\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:12.380669 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.380627    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vgjv\" (UniqueName: \"kubernetes.io/projected/e5c2b8fb-331d-4df2-ad30-13549d8a660c-kube-api-access-2vgjv\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:12.382608 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.382588    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istio-token\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:12.387617 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.387593    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:23:12.475410 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.468384    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-istio-envoy\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:12.475410 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.468503    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-credential-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:12.475410 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.468638    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1a1157d-88ea-448f-931d-9528c5df5684-istio-token\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:12.475410 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.468719    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1a1157d-88ea-448f-931d-9528c5df5684-istio-podinfo\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:12.475410 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.468771    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-workload-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:12.475410 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.468799    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1a1157d-88ea-448f-931d-9528c5df5684-istiod-ca-cert\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:12.475410 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.468868    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-workload-certs\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:12.475410 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.468906    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbxrn\" (UniqueName: \"kubernetes.io/projected/b1a1157d-88ea-448f-931d-9528c5df5684-kube-api-access-zbxrn\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:12.475410 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.469031    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-istio-data\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:12.475410 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.469452    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-credential-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:12.475410 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.470966    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1a1157d-88ea-448f-931d-9528c5df5684-istiod-ca-cert\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:12.475410 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.471177    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-workload-certs\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:12.475410 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.471411    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-workload-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:12.475410 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.471580    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-istio-data\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:12.475410 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.474683    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-istio-envoy\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:12.475410 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.475348    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1a1157d-88ea-448f-931d-9528c5df5684-istio-podinfo\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:12.488190 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.488138    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1a1157d-88ea-448f-931d-9528c5df5684-istio-token\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:12.493081 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.493016    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbxrn\" (UniqueName: \"kubernetes.io/projected/b1a1157d-88ea-448f-931d-9528c5df5684-kube-api-access-zbxrn\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:12.590187 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.589925    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:12.596762 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.596556    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:12.790349 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.790311    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"]
Oct 31 23:23:12.807698 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.807489    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"]
Oct 31 23:23:12.813316 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.813274    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:23:12.813467 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.813431    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:23:12.813543 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.813520    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:23:12.813603 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.813565    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:23:12.813912 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.813885    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:23:12.814066 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:12.814048    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:23:13.203470 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:13.203439    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"]
Oct 31 23:23:13.206847 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:13.206798    4301 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 31 23:23:13.206988 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:13.206863    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd" podUID="34836e36-7a9c-4072-a555-970e2eb17c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 31 23:23:13.230410 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:13.228538    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:23:13.230410 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:13.228630    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:23:13.230410 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:13.228674    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:23:13.230410 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:13.229434    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"]
Oct 31 23:23:13.230410 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:13.230062    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:23:13.230410 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:13.230134    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:23:13.230410 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:13.230174    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:23:13.309089 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:13.309031    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf" event={"ID":"2b5f9bbb-897d-44dc-9eb8-5887091b4d64","Type":"ContainerStarted","Data":"c46aff3eccf6248c32249539e8a9da8fd01a605fcafdda31143efae0804002ea"}
Oct 31 23:23:13.309254 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:13.309144    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf" event={"ID":"2b5f9bbb-897d-44dc-9eb8-5887091b4d64","Type":"ContainerStarted","Data":"5b10574ca343e10693f9ff4e61da84299bb5bc2de587da3c05fb850c250419d6"}
Oct 31 23:23:13.325452 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:13.325341    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2" event={"ID":"bba885b3-4cbf-4677-bef9-94f492ca8794","Type":"ContainerStarted","Data":"004c101df93af2590ce5f0f449ca6129709bb62dbf8db2245e80369b76c4b0e0"}
Oct 31 23:23:13.325607 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:13.325457    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2" event={"ID":"bba885b3-4cbf-4677-bef9-94f492ca8794","Type":"ContainerStarted","Data":"1177c510debc0a4a27c849e2d60362d7cdd27f974e52dab9b5553c86688ef986"}
Oct 31 23:23:13.327495 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:13.327460    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:23:13.410826 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:13.393490    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:23:13.421228 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:13.418451    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf" podStartSLOduration=2.418420989 podStartE2EDuration="2.418420989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:23:13.39253564 +0000 UTC m=+2012.139725153" watchObservedRunningTime="2026-10-31 23:23:13.418420989 +0000 UTC m=+2012.165610494"
Oct 31 23:23:13.485399 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:13.483706    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74a97819-25d1-43c8-b104-ee21999c5f68" path="/var/lib/kubelet/pods/74a97819-25d1-43c8-b104-ee21999c5f68/volumes"
Oct 31 23:23:13.485399 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:13.484387    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7560a60-cbba-43ba-81a1-946c57663b80" path="/var/lib/kubelet/pods/f7560a60-cbba-43ba-81a1-946c57663b80/volumes"
Oct 31 23:23:14.212420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:14.208989    4301 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 31 23:23:14.212420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:14.209048    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd" podUID="34836e36-7a9c-4072-a555-970e2eb17c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 31 23:23:14.374385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:14.339951    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 31 23:23:14.407546 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:14.407492    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2" podUID="bba885b3-4cbf-4677-bef9-94f492ca8794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 31 23:23:14.408279 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:14.408255    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 23:23:14.409181 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:14.409152    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf" podUID="2b5f9bbb-897d-44dc-9eb8-5887091b4d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 23:23:14.410513 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:14.410489    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh" event={"ID":"b1a1157d-88ea-448f-931d-9528c5df5684","Type":"ContainerStarted","Data":"fa51886660aad72a3e8fab6906996840346df64773a5627edb1abfd47c6a2609"}
Oct 31 23:23:14.410702 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:14.410683    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh" event={"ID":"b1a1157d-88ea-448f-931d-9528c5df5684","Type":"ContainerStarted","Data":"5e9ad06e2f7a735235737837f7071704e0a9eafa0b6ec9f7787c160bbb2d35c8"}
Oct 31 23:23:14.432165 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:14.432130    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz" event={"ID":"e5c2b8fb-331d-4df2-ad30-13549d8a660c","Type":"ContainerStarted","Data":"222190fd321ae01f361feeaa7c2fe1704ccc18b1be65e3f88b25bbb3ccc70e1f"}
Oct 31 23:23:14.433270 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:14.433244    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz" event={"ID":"e5c2b8fb-331d-4df2-ad30-13549d8a660c","Type":"ContainerStarted","Data":"e0a4965898830da0e2e558d0e37193b4fa76338aa1a9b654567e72147f670545"}
Oct 31 23:23:14.509432 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:14.509144    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh" podStartSLOduration=3.5091245410000003 podStartE2EDuration="3.509124541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:23:14.486840796 +0000 UTC m=+2013.234030295" watchObservedRunningTime="2026-10-31 23:23:14.509124541 +0000 UTC m=+2013.256314067"
Oct 31 23:23:14.534383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:14.511718    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2" podStartSLOduration=3.511700345 podStartE2EDuration="3.511700345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:23:13.422103497 +0000 UTC m=+2012.169293005" watchObservedRunningTime="2026-10-31 23:23:14.511700345 +0000 UTC m=+2013.258889851"
Oct 31 23:23:14.599457 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:14.599420    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:14.613429 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:14.613403    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:14.613676 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:14.612526    4301 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 31 23:23:14.613847 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:14.613824    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz" podUID="e5c2b8fb-331d-4df2-ad30-13549d8a660c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 31 23:23:14.613979 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:14.613884    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz" podStartSLOduration=3.6138648 podStartE2EDuration="3.6138648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:23:14.54612577 +0000 UTC m=+2013.293315277" watchObservedRunningTime="2026-10-31 23:23:14.6138648 +0000 UTC m=+2013.361054307"
Oct 31 23:23:14.625896 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:14.613258    4301 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 23:23:14.626323 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:14.626291    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh" podUID="b1a1157d-88ea-448f-931d-9528c5df5684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 23:23:15.230381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:15.229731    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:23:15.234384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:15.231545    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:23:15.248093 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:15.248061    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:23:15.259550 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:15.259504    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 23:23:15.331685 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:15.331638    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 31 23:23:15.331941 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:15.331906    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2" podUID="bba885b3-4cbf-4677-bef9-94f492ca8794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 31 23:23:15.379221 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:15.367531    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 23:23:15.410392 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:15.407564    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 23:23:15.410392 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:15.407621    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf" podUID="2b5f9bbb-897d-44dc-9eb8-5887091b4d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 23:23:15.465123 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:15.465073    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:15.545465 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:15.537197    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:15.892467 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:15.830259    4301 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 31 23:23:15.892467 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:15.830328    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz" podUID="e5c2b8fb-331d-4df2-ad30-13549d8a660c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 31 23:23:15.892467 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:15.830457    4301 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 23:23:15.892467 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:15.830491    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh" podUID="b1a1157d-88ea-448f-931d-9528c5df5684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 23:23:15.928466 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:15.928343    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:15.969448 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:15.968154    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"]
Oct 31 23:23:15.998382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:15.994226    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:16.042381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.015910    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:23:16.086385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.061514    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-836c99-openshift-default-dockercfg-zn6fh\""
Oct 31 23:23:16.132351 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.132308    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-credential-socket\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:23:16.132556 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.132454    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-workload-certs\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:23:16.132556 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.132482    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xq92\" (UniqueName: \"kubernetes.io/projected/3cab9805-e8f6-4028-9418-d522053546db-kube-api-access-2xq92\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:23:16.132556 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.132507    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-istio-envoy\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:23:16.132556 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.132551    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cab9805-e8f6-4028-9418-d522053546db-istio-token\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:23:16.132787 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.132725    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cab9805-e8f6-4028-9418-d522053546db-istiod-ca-cert\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:23:16.168459 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.132904    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-workload-socket\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:23:16.168459 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.132935    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-istio-data\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:23:16.168459 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.133169    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cab9805-e8f6-4028-9418-d522053546db-istio-podinfo\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:23:16.262434 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.210442    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"]
Oct 31 23:23:16.270921 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.270875    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-credential-socket\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:23:16.284813 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.284764    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-workload-certs\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:23:16.334771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.314862    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xq92\" (UniqueName: \"kubernetes.io/projected/3cab9805-e8f6-4028-9418-d522053546db-kube-api-access-2xq92\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:23:16.334771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.314923    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-istio-envoy\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:23:16.334771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.314978    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cab9805-e8f6-4028-9418-d522053546db-istio-token\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:23:16.334771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.310021    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-workload-certs\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:23:16.334771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.272030    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-credential-socket\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:23:16.334771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.317522    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cab9805-e8f6-4028-9418-d522053546db-istiod-ca-cert\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:23:16.334771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.321447    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cab9805-e8f6-4028-9418-d522053546db-istiod-ca-cert\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:23:16.334771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.321683    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-workload-socket\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:23:16.334771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.321716    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-istio-data\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:23:16.334771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.321873    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cab9805-e8f6-4028-9418-d522053546db-istio-podinfo\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:23:16.334771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.322655    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-workload-socket\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:23:16.334771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.322909    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-istio-data\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:23:16.334771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.323447    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 31 23:23:16.334771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.323496    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2" podUID="bba885b3-4cbf-4677-bef9-94f492ca8794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 31 23:23:16.334771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.334031    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:16.371568 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.369534    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:16.371568 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.370700    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:23:16.390157 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.389739    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 23:23:16.390157 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.389797    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf" podUID="2b5f9bbb-897d-44dc-9eb8-5887091b4d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 23:23:16.468744 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.468641    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:16.478449 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.478406    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:16.479100 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.479082    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:23:16.496438 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.482913    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cab9805-e8f6-4028-9418-d522053546db-istio-podinfo\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:23:16.496438 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.483533    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xq92\" (UniqueName: \"kubernetes.io/projected/3cab9805-e8f6-4028-9418-d522053546db-kube-api-access-2xq92\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:23:16.572065 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.572003    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:16.573493 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.573270    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:23:16.597423 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.595136    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cab9805-e8f6-4028-9418-d522053546db-istio-token\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:23:16.608643 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.608290    4301 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 23:23:16.608643 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.608349    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh" podUID="b1a1157d-88ea-448f-931d-9528c5df5684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 23:23:16.872500 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.867572    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded"
Oct 31 23:23:16.933378 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.872700    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-istio-envoy\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:23:17.095902 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.954223    4301 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 31 23:23:17.095902 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:16.954284    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz" podUID="e5c2b8fb-331d-4df2-ad30-13549d8a660c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 31 23:23:17.111382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:17.098636    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:23:17.190815 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:17.189294    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:17.231397 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:17.217961    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:23:17.383015 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:17.283457    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:17.383015 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:17.299580    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:23:17.383015 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:17.303121    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded"
Oct 31 23:23:17.383015 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:17.303609    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 23:23:17.383015 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:17.347812    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:17.383015 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:17.348531    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:23:17.383015 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:17.375958    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:17.392497 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:17.392456    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Oct 31 23:23:17.437853 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:17.428597    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 23:23:17.553383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:17.543784    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:17.603385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:17.595820    4301 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 31 23:23:17.612202 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:17.612155    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz" podUID="e5c2b8fb-331d-4df2-ad30-13549d8a660c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 31 23:23:17.671530 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:17.671472    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:17.672465 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:17.672435    4301 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 23:23:17.709918 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:17.709832    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh" podUID="b1a1157d-88ea-448f-931d-9528c5df5684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 23:23:17.710452 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:17.672643    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:17.710968 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:17.710945    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 31 23:23:17.737044 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:17.737009    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2" podUID="bba885b3-4cbf-4677-bef9-94f492ca8794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 31 23:23:17.757780 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:17.741105    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 23:23:17.757780 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:17.741158    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf" podUID="2b5f9bbb-897d-44dc-9eb8-5887091b4d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 23:23:18.124399 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:18.123725    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:23:18.319523 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:18.319421    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:18.320912 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:18.320879    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:18.371768 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:18.334432    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:18.403208 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:18.377305    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:23:18.446836 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:18.446323    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:18.446836 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:18.446681    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:23:18.446836 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:18.468093    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:18.446836 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:18.468548    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:18.499647 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:18.477646    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:18.524824 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:18.514924    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 31 23:23:18.524824 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:18.514978    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2" podUID="bba885b3-4cbf-4677-bef9-94f492ca8794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 31 23:23:18.524824 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:18.516284    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 23:23:18.541406 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:18.536683    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf" podUID="2b5f9bbb-897d-44dc-9eb8-5887091b4d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 23:23:18.780762 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:18.685629    4301 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 23:23:18.780762 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:18.685683    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh" podUID="b1a1157d-88ea-448f-931d-9528c5df5684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 23:23:18.780762 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:18.685887    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:23:18.780762 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:18.686592    4301 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 31 23:23:18.780762 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:18.686631    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz" podUID="e5c2b8fb-331d-4df2-ad30-13549d8a660c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 31 23:23:19.413786 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:19.413747    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 31 23:23:19.474383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:19.468717    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:19.474383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:19.468918    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 23:23:19.474383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:19.470443    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:19.474383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:19.470900    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 23:23:19.518393 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:19.499438    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 31 23:23:19.518393 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:19.499501    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2" podUID="bba885b3-4cbf-4677-bef9-94f492ca8794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 31 23:23:19.525175 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:19.525142    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 23:23:19.541413 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:19.525429    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf" podUID="2b5f9bbb-897d-44dc-9eb8-5887091b4d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 23:23:19.541721 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:19.530434    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:19.625889 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:19.612031    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 23:23:19.679782 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:19.612085    4301 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 31 23:23:19.701132 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:19.701087    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz" podUID="e5c2b8fb-331d-4df2-ad30-13549d8a660c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 31 23:23:19.701377 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:19.680982    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 31 23:23:19.704511 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:19.704487    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:23:19.705695 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:19.670856    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:23:19.708605 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:19.705985    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:23:19.709121 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:19.709092    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:23:20.101963 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:19.649921    4301 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 23:23:20.103999 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:20.103969    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh" podUID="b1a1157d-88ea-448f-931d-9528c5df5684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 23:23:20.104206 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:20.043883    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"]
Oct 31 23:23:20.492447 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:20.475458    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 31 23:23:20.538426 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:20.512539    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 31 23:23:20.541504 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:20.527763    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 23:23:20.541504 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:20.539213    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 23:23:20.541504 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:20.541376    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf" podUID="2b5f9bbb-897d-44dc-9eb8-5887091b4d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 23:23:20.541504 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:20.538396    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 31 23:23:20.541504 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:20.541447    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2" podUID="bba885b3-4cbf-4677-bef9-94f492ca8794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 31 23:23:20.542296 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:20.541968    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:20.572419 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:20.572270    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:20.618449 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:20.596514    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:20.618449 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:20.617533    4301 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 31 23:23:20.618449 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:20.617591    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz" podUID="e5c2b8fb-331d-4df2-ad30-13549d8a660c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 31 23:23:20.644421 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:20.619305    4301 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 23:23:20.644421 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:20.619351    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh" podUID="b1a1157d-88ea-448f-931d-9528c5df5684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 23:23:20.720436 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:20.709436    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 31 23:23:20.720436 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:20.709900    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:23:20.830410 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:20.830368    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7" event={"ID":"3cab9805-e8f6-4028-9418-d522053546db","Type":"ContainerStarted","Data":"90b3d128e8862db51763609c52d7aaa70fe38f6e109d16997514ddd0effc11c3"}
Oct 31 23:23:21.006824 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:20.982666    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:23:21.008319 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:21.008277    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:23:21.107526 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:21.107479    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:21.365887 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:21.365832    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:21.534598 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:21.534484    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 23:23:21.706399 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:21.568694    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:21.721385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:21.707503    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:22.137403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:21.756686    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:22.137403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:21.867870    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:22.137403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:21.904461    4301 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 31 23:23:22.137403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:22.128775    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz" podUID="e5c2b8fb-331d-4df2-ad30-13549d8a660c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 31 23:23:22.137403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:21.904976    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 23:23:22.137403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:22.128846    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf" podUID="2b5f9bbb-897d-44dc-9eb8-5887091b4d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 23:23:22.137403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:21.914899    4301 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 23:23:22.137403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:22.128906    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh" podUID="b1a1157d-88ea-448f-931d-9528c5df5684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 23:23:22.137403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:22.014630    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 31 23:23:22.137403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:22.128953    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2" podUID="bba885b3-4cbf-4677-bef9-94f492ca8794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 31 23:23:22.174385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:22.158862    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:22.331539 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:22.331489    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:22.398385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:22.395993    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 23:23:22.398385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:22.396070    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf" podUID="2b5f9bbb-897d-44dc-9eb8-5887091b4d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 23:23:22.398954 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:22.398917    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:22.556569 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:22.556383    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 31 23:23:22.557147 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:22.557117    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2" podUID="bba885b3-4cbf-4677-bef9-94f492ca8794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 31 23:23:22.585545 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:22.585510    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 23:23:22.590445 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:22.585448    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:22.592220 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:22.592180    4301 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 31 23:23:22.592468 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:22.592431    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz" podUID="e5c2b8fb-331d-4df2-ad30-13549d8a660c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 31 23:23:22.592664 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:23:22.577527    4301 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.018s"
Oct 31 23:23:22.592955 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:22.592930    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:23:22.593684 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:22.593423    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:23:22.606729 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:22.606699    4301 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 23:23:22.606945 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:22.606919    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh" podUID="b1a1157d-88ea-448f-931d-9528c5df5684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 23:23:22.631206 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:22.630864    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:23:22.779573 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:22.779540    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7" event={"ID":"3cab9805-e8f6-4028-9418-d522053546db","Type":"ContainerStarted","Data":"5b3f84930779e3fd10a24319161428699fe81315605730ff462000f88125318f"}
Oct 31 23:23:22.943392 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:22.933558    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"]
Oct 31 23:23:22.969400 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:22.943890    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz" podUID="a3ef6611-0d56-41c3-9357-81e33a2fd67a" containerName="istio-proxy" containerID="cri-o://803872df6429ac59bec602fc843243af9aea7bd08cde7d51fcfaca6473687042" gracePeriod=30
Oct 31 23:23:23.143570 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:23.143529    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 23:23:23.203437 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:23.143495    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 23:23:23.203437 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:23.150049    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:23:23.203437 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:23.151720    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 31 23:23:23.203437 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:23.162405    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:23.230774 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:23.224842    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:23:23.230774 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:23.229835    4301 patch_prober.go:29] interesting pod/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 31 23:23:23.230774 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:23.229886    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7" podUID="3cab9805-e8f6-4028-9418-d522053546db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 31 23:23:23.381920 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:23.381883    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 31 23:23:23.382221 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:23.382190    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2" podUID="bba885b3-4cbf-4677-bef9-94f492ca8794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 31 23:23:23.412391 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:23.385223    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:23.428436 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:23.416220    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 23:23:23.428436 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:23.416274    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf" podUID="2b5f9bbb-897d-44dc-9eb8-5887091b4d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 23:23:23.568515 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:23.557508    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:23.588273 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:23.588220    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:23.590405 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:23.590352    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:23.600485 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:23.590903    4301 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 31 23:23:23.623687 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:23.608005    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz" podUID="e5c2b8fb-331d-4df2-ad30-13549d8a660c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 31 23:23:23.643460 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:23.643422    4301 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 23:23:23.643668 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:23.643486    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh" podUID="b1a1157d-88ea-448f-931d-9528c5df5684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 23:23:23.810793 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:23.810710    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:23:23.856629 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:23.842870    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7" podStartSLOduration=8.842850984 podStartE2EDuration="8.842850984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:23:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:23:22.817220442 +0000 UTC m=+2021.564409953" watchObservedRunningTime="2026-10-31 23:23:23.842850984 +0000 UTC m=+2022.590040490"
Oct 31 23:23:24.147524 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:24.147475    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:24.150776 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:24.150453    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:24.164393 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:24.154501    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 31 23:23:24.186613 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:24.170453    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:24.227411 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:24.205496    4301 patch_prober.go:29] interesting pod/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 31 23:23:24.227411 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:24.205558    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7" podUID="3cab9805-e8f6-4028-9418-d522053546db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 31 23:23:24.249555 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:24.249257    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1faf30d6-dvscm"]
Oct 31 23:23:24.322786 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:24.322751    4301 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 31 23:23:24.323105 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:24.323076    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2" podUID="bba885b3-4cbf-4677-bef9-94f492ca8794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 31 23:23:24.334390 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:24.324454    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1faf30d6-dvscm"
Oct 31 23:23:24.346137 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:24.346108    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1faf30d6-dvscm"]
Oct 31 23:23:24.358755 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:24.353724    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 23:23:24.358755 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:24.353862    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\""
Oct 31 23:23:24.416462 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:24.415434    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 31 23:23:24.530381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:24.508129    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x47mz\" (UniqueName: \"kubernetes.io/projected/b05705b9-6c2f-4c7f-9d58-e832c56afde5-kube-api-access-x47mz\") pod \"maas-api-revoke-sub-1faf30d6-dvscm\" (UID: \"b05705b9-6c2f-4c7f-9d58-e832c56afde5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1faf30d6-dvscm"
Oct 31 23:23:24.530381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:24.508349    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b05705b9-6c2f-4c7f-9d58-e832c56afde5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1faf30d6-dvscm\" (UID: \"b05705b9-6c2f-4c7f-9d58-e832c56afde5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1faf30d6-dvscm"
Oct 31 23:23:24.530381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:24.508468    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b05705b9-6c2f-4c7f-9d58-e832c56afde5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1faf30d6-dvscm\" (UID: \"b05705b9-6c2f-4c7f-9d58-e832c56afde5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1faf30d6-dvscm"
Oct 31 23:23:24.556905 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:24.556862    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:24.616128 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:24.612768    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:23:24.616128 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:24.614566    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:23:24.616128 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:24.614760    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:24.619474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:24.619040    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:23:24.668658 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:24.668518    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 23:23:24.679656 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:24.679596    4301 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 31 23:23:24.810129 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:24.810083    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz" podUID="e5c2b8fb-331d-4df2-ad30-13549d8a660c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 31 23:23:24.810985 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:24.734927    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x47mz\" (UniqueName: \"kubernetes.io/projected/b05705b9-6c2f-4c7f-9d58-e832c56afde5-kube-api-access-x47mz\") pod \"maas-api-revoke-sub-1faf30d6-dvscm\" (UID: \"b05705b9-6c2f-4c7f-9d58-e832c56afde5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1faf30d6-dvscm"
Oct 31 23:23:24.811600 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:24.811570    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b05705b9-6c2f-4c7f-9d58-e832c56afde5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1faf30d6-dvscm\" (UID: \"b05705b9-6c2f-4c7f-9d58-e832c56afde5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1faf30d6-dvscm"
Oct 31 23:23:24.811897 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:24.811880    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b05705b9-6c2f-4c7f-9d58-e832c56afde5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1faf30d6-dvscm\" (UID: \"b05705b9-6c2f-4c7f-9d58-e832c56afde5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1faf30d6-dvscm"
Oct 31 23:23:24.845403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:24.825959    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b05705b9-6c2f-4c7f-9d58-e832c56afde5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1faf30d6-dvscm\" (UID: \"b05705b9-6c2f-4c7f-9d58-e832c56afde5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1faf30d6-dvscm"
Oct 31 23:23:24.915411 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:24.906688    4301 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 23:23:24.915411 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:24.906752    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh" podUID="b1a1157d-88ea-448f-931d-9528c5df5684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 23:23:24.957758 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:24.957663    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:24.993121 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:24.993083    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b05705b9-6c2f-4c7f-9d58-e832c56afde5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1faf30d6-dvscm\" (UID: \"b05705b9-6c2f-4c7f-9d58-e832c56afde5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1faf30d6-dvscm"
Oct 31 23:23:25.044770 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:25.044739    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:23:25.120664 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:25.120556    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:23:25.205474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:25.166686    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:23:25.304458 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:25.298658    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:23:25.304458 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:25.298730    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:23:25.304458 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:25.220192    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 31 23:23:25.304458 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:25.253100    4301 patch_prober.go:29] interesting pod/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 31 23:23:25.304458 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:25.301930    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7" podUID="3cab9805-e8f6-4028-9418-d522053546db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 31 23:23:25.402690 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:25.401264    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x47mz\" (UniqueName: \"kubernetes.io/projected/b05705b9-6c2f-4c7f-9d58-e832c56afde5-kube-api-access-x47mz\") pod \"maas-api-revoke-sub-1faf30d6-dvscm\" (UID: \"b05705b9-6c2f-4c7f-9d58-e832c56afde5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1faf30d6-dvscm"
Oct 31 23:23:25.421897 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:25.419759    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 31 23:23:25.515726 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:25.515682    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:25.517595 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:25.517553    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:23:25.517859 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:25.517843    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:23:25.517994 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:25.517984    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:23:25.519729 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:25.519706    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:23:25.522015 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:25.521988    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:23:25.595433 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:25.595268    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1faf30d6-dvscm"
Oct 31 23:23:25.830392 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:25.812927    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:25.830392 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:25.824506    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:25.830392 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:25.829948    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:26.036393 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:26.034814    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:26.036619 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:26.036563    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:26.052612 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:26.040387    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:26.196340 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:26.196300    4301 patch_prober.go:29] interesting pod/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 31 23:23:26.196585 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:26.196397    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7" podUID="3cab9805-e8f6-4028-9418-d522053546db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 31 23:23:26.636120 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:26.608693    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1faf30d6-dvscm"]
Oct 31 23:23:27.194795 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:27.194757    4301 patch_prober.go:29] interesting pod/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 31 23:23:27.195889 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:27.195851    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7" podUID="3cab9805-e8f6-4028-9418-d522053546db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 31 23:23:27.289907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:27.289820    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1faf30d6-dvscm" event={"ID":"b05705b9-6c2f-4c7f-9d58-e832c56afde5","Type":"ContainerStarted","Data":"9d161b577b363c777ac19fbb1c8b7798ad60c2bdf72816cd91210c1be7ac70eb"}
Oct 31 23:23:28.206590 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.206550    4301 patch_prober.go:29] interesting pod/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 31 23:23:28.207064 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.206614    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7" podUID="3cab9805-e8f6-4028-9418-d522053546db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 31 23:23:28.311986 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.311953    4301 generic.go:356] "Generic (PLEG): container finished" podID="a3ef6611-0d56-41c3-9357-81e33a2fd67a" containerID="803872df6429ac59bec602fc843243af9aea7bd08cde7d51fcfaca6473687042" exitCode=0
Oct 31 23:23:28.312170 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.312156    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz" event={"ID":"a3ef6611-0d56-41c3-9357-81e33a2fd67a","Type":"ContainerDied","Data":"803872df6429ac59bec602fc843243af9aea7bd08cde7d51fcfaca6473687042"}
Oct 31 23:23:28.312218 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.312205    4301 scope.go:122] "RemoveContainer" containerID="cb73ef2b0577265122157fc5171ab5630a7402bac27008f6b293da6d3ba371d3"
Oct 31 23:23:28.315767 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.315741    4301 generic.go:356] "Generic (PLEG): container finished" podID="b05705b9-6c2f-4c7f-9d58-e832c56afde5" containerID="286b3fb755963c623ecffb17c1e45670fdac2bee61228b02e4d1f85eb9087478" exitCode=0
Oct 31 23:23:28.315884 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.315798    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1faf30d6-dvscm" event={"ID":"b05705b9-6c2f-4c7f-9d58-e832c56afde5","Type":"ContainerDied","Data":"286b3fb755963c623ecffb17c1e45670fdac2bee61228b02e4d1f85eb9087478"}
Oct 31 23:23:28.525185 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.525151    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:23:28.623271 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.623242    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-workload-socket\") pod \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") "
Oct 31 23:23:28.623490 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.623310    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istiod-ca-cert\") pod \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") "
Oct 31 23:23:28.623490 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.623344    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istio-data\") pod \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") "
Oct 31 23:23:28.623732 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.623707    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istio-envoy\") pod \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") "
Oct 31 23:23:28.623841 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.623822    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-workload-certs\") pod \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") "
Oct 31 23:23:28.623986 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.623967    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ef6611-0d56-41c3-9357-81e33a2fd67a-kube-api-access-q6cfr\" (UniqueName: \"kubernetes.io/projected/a3ef6611-0d56-41c3-9357-81e33a2fd67a-kube-api-access-q6cfr\") pod \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") "
Oct 31 23:23:28.624120 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.624066    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-credential-socket\") pod \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") "
Oct 31 23:23:28.624207 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.624192    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istio-podinfo\") pod \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") "
Oct 31 23:23:28.624266 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.624237    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istio-token\" (UniqueName: \"kubernetes.io/projected/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istio-token\") pod \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\" (UID: \"a3ef6611-0d56-41c3-9357-81e33a2fd67a\") "
Oct 31 23:23:28.624419 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.624376    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-workload-socket" pod "a3ef6611-0d56-41c3-9357-81e33a2fd67a" (UID: "a3ef6611-0d56-41c3-9357-81e33a2fd67a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:28.625044 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.625021    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:28.625200 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.625178    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-workload-certs" pod "a3ef6611-0d56-41c3-9357-81e33a2fd67a" (UID: "a3ef6611-0d56-41c3-9357-81e33a2fd67a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:28.625519 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.625497    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istio-data" pod "a3ef6611-0d56-41c3-9357-81e33a2fd67a" (UID: "a3ef6611-0d56-41c3-9357-81e33a2fd67a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:28.625862 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.625848    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istiod-ca-cert" pod "a3ef6611-0d56-41c3-9357-81e33a2fd67a" (UID: "a3ef6611-0d56-41c3-9357-81e33a2fd67a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:23:28.626114 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.626103    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-credential-socket" pod "a3ef6611-0d56-41c3-9357-81e33a2fd67a" (UID: "a3ef6611-0d56-41c3-9357-81e33a2fd67a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:28.627694 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.627661    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istio-token" pod "a3ef6611-0d56-41c3-9357-81e33a2fd67a" (UID: "a3ef6611-0d56-41c3-9357-81e33a2fd67a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:23:28.628239 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.628214    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istio-envoy" pod "a3ef6611-0d56-41c3-9357-81e33a2fd67a" (UID: "a3ef6611-0d56-41c3-9357-81e33a2fd67a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:28.655574 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.653559    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ef6611-0d56-41c3-9357-81e33a2fd67a-kube-api-access-q6cfr" pod "a3ef6611-0d56-41c3-9357-81e33a2fd67a" (UID: "a3ef6611-0d56-41c3-9357-81e33a2fd67a"). InnerVolumeSpecName "kube-api-access-q6cfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:23:28.655574 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.654156    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istio-podinfo" pod "a3ef6611-0d56-41c3-9357-81e33a2fd67a" (UID: "a3ef6611-0d56-41c3-9357-81e33a2fd67a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:23:28.726558 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.726524    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:28.726558 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.726554    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:28.726787 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.726567    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6cfr\" (UniqueName: \"kubernetes.io/projected/a3ef6611-0d56-41c3-9357-81e33a2fd67a-kube-api-access-q6cfr\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:28.726787 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.726580    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:28.726787 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.726593    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:28.726787 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.726604    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:28.726787 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.726615    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:28.726787 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:28.726627    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ef6611-0d56-41c3-9357-81e33a2fd67a-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:29.190440 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:29.190404    4301 patch_prober.go:29] interesting pod/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 31 23:23:29.190606 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:29.190458    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7" podUID="3cab9805-e8f6-4028-9418-d522053546db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 31 23:23:29.324691 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:29.324663    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"
Oct 31 23:23:29.325176 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:29.324701    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz" event={"ID":"a3ef6611-0d56-41c3-9357-81e33a2fd67a","Type":"ContainerDied","Data":"556cf16412aee045356afec034ce8b88ea9a37307a66cca5f57414a2a2c5bbca"}
Oct 31 23:23:29.325176 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:29.324784    4301 scope.go:122] "RemoveContainer" containerID="803872df6429ac59bec602fc843243af9aea7bd08cde7d51fcfaca6473687042"
Oct 31 23:23:29.628775 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:29.609852    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"]
Oct 31 23:23:29.705347 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:29.705256    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-36a15fc5-openshift-default-899fcfc7c-jpssz"]
Oct 31 23:23:29.906509 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:29.906469    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1deb6-mmcrn"]
Oct 31 23:23:29.907504 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:29.907466    4301 cpu_manager.go:414] "containerMap: removing container" podUID="a3ef6611-0d56-41c3-9357-81e33a2fd67a" containerName="istio-proxy"
Oct 31 23:23:29.907504 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:29.907491    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3ef6611-0d56-41c3-9357-81e33a2fd67a" containerName="istio-proxy"
Oct 31 23:23:29.907709 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:29.907510    4301 cpu_manager.go:414] "containerMap: removing container" podUID="a3ef6611-0d56-41c3-9357-81e33a2fd67a" containerName="istio-proxy"
Oct 31 23:23:29.907709 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:29.907518    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3ef6611-0d56-41c3-9357-81e33a2fd67a" containerName="istio-proxy"
Oct 31 23:23:29.907709 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:29.907649    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3ef6611-0d56-41c3-9357-81e33a2fd67a" containerName="istio-proxy"
Oct 31 23:23:29.907709 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:29.907662    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3ef6611-0d56-41c3-9357-81e33a2fd67a" containerName="istio-proxy"
Oct 31 23:23:29.913951 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:29.913921    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1deb6-mmcrn"]
Oct 31 23:23:29.914106 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:29.914033    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1deb6-mmcrn"
Oct 31 23:23:29.967376 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:29.967280    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa97ea3-555d-4b46-9b55-7e55c7481e0b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a1deb6-mmcrn\" (UID: \"7aa97ea3-555d-4b46-9b55-7e55c7481e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1deb6-mmcrn"
Oct 31 23:23:29.967541 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:29.967519    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa97ea3-555d-4b46-9b55-7e55c7481e0b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a1deb6-mmcrn\" (UID: \"7aa97ea3-555d-4b46-9b55-7e55c7481e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1deb6-mmcrn"
Oct 31 23:23:29.967597 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:29.967576    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbmvc\" (UniqueName: \"kubernetes.io/projected/7aa97ea3-555d-4b46-9b55-7e55c7481e0b-kube-api-access-zbmvc\") pod \"maas-api-revoke-keys-e2e-api-a-a1deb6-mmcrn\" (UID: \"7aa97ea3-555d-4b46-9b55-7e55c7481e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1deb6-mmcrn"
Oct 31 23:23:30.069584 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:30.069546    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa97ea3-555d-4b46-9b55-7e55c7481e0b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a1deb6-mmcrn\" (UID: \"7aa97ea3-555d-4b46-9b55-7e55c7481e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1deb6-mmcrn"
Oct 31 23:23:30.069801 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:30.069779    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa97ea3-555d-4b46-9b55-7e55c7481e0b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a1deb6-mmcrn\" (UID: \"7aa97ea3-555d-4b46-9b55-7e55c7481e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1deb6-mmcrn"
Oct 31 23:23:30.069897 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:30.069878    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbmvc\" (UniqueName: \"kubernetes.io/projected/7aa97ea3-555d-4b46-9b55-7e55c7481e0b-kube-api-access-zbmvc\") pod \"maas-api-revoke-keys-e2e-api-a-a1deb6-mmcrn\" (UID: \"7aa97ea3-555d-4b46-9b55-7e55c7481e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1deb6-mmcrn"
Oct 31 23:23:30.070415 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:30.070394    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa97ea3-555d-4b46-9b55-7e55c7481e0b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a1deb6-mmcrn\" (UID: \"7aa97ea3-555d-4b46-9b55-7e55c7481e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1deb6-mmcrn"
Oct 31 23:23:30.072698 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:30.072670    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa97ea3-555d-4b46-9b55-7e55c7481e0b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a1deb6-mmcrn\" (UID: \"7aa97ea3-555d-4b46-9b55-7e55c7481e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1deb6-mmcrn"
Oct 31 23:23:30.083206 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:30.083178    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbmvc\" (UniqueName: \"kubernetes.io/projected/7aa97ea3-555d-4b46-9b55-7e55c7481e0b-kube-api-access-zbmvc\") pod \"maas-api-revoke-keys-e2e-api-a-a1deb6-mmcrn\" (UID: \"7aa97ea3-555d-4b46-9b55-7e55c7481e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1deb6-mmcrn"
Oct 31 23:23:30.114820 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:30.114792    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1faf30d6-dvscm"
Oct 31 23:23:30.171132 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:30.171086    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b05705b9-6c2f-4c7f-9d58-e832c56afde5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b05705b9-6c2f-4c7f-9d58-e832c56afde5-maas-api-xks-ca\") pod \"b05705b9-6c2f-4c7f-9d58-e832c56afde5\" (UID: \"b05705b9-6c2f-4c7f-9d58-e832c56afde5\") "
Oct 31 23:23:30.171461 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:30.171445    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b05705b9-6c2f-4c7f-9d58-e832c56afde5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b05705b9-6c2f-4c7f-9d58-e832c56afde5-maas-api-openshift-ca\") pod \"b05705b9-6c2f-4c7f-9d58-e832c56afde5\" (UID: \"b05705b9-6c2f-4c7f-9d58-e832c56afde5\") "
Oct 31 23:23:30.171779 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:30.171764    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b05705b9-6c2f-4c7f-9d58-e832c56afde5-kube-api-access-x47mz\" (UniqueName: \"kubernetes.io/projected/b05705b9-6c2f-4c7f-9d58-e832c56afde5-kube-api-access-x47mz\") pod \"b05705b9-6c2f-4c7f-9d58-e832c56afde5\" (UID: \"b05705b9-6c2f-4c7f-9d58-e832c56afde5\") "
Oct 31 23:23:30.172166 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:30.171821    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b05705b9-6c2f-4c7f-9d58-e832c56afde5-maas-api-openshift-ca" pod "b05705b9-6c2f-4c7f-9d58-e832c56afde5" (UID: "b05705b9-6c2f-4c7f-9d58-e832c56afde5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:23:30.173529 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:30.173506    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b05705b9-6c2f-4c7f-9d58-e832c56afde5-maas-api-openshift-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:30.176015 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:30.175986    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b05705b9-6c2f-4c7f-9d58-e832c56afde5-maas-api-xks-ca" pod "b05705b9-6c2f-4c7f-9d58-e832c56afde5" (UID: "b05705b9-6c2f-4c7f-9d58-e832c56afde5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:23:30.176161 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:30.176140    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b05705b9-6c2f-4c7f-9d58-e832c56afde5-kube-api-access-x47mz" pod "b05705b9-6c2f-4c7f-9d58-e832c56afde5" (UID: "b05705b9-6c2f-4c7f-9d58-e832c56afde5"). InnerVolumeSpecName "kube-api-access-x47mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:23:30.189997 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:30.189960    4301 patch_prober.go:29] interesting pod/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 31 23:23:30.190096 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:30.190033    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7" podUID="3cab9805-e8f6-4028-9418-d522053546db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 31 23:23:30.269471 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:30.267229    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1deb6-mmcrn"
Oct 31 23:23:30.275725 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:30.275697    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x47mz\" (UniqueName: \"kubernetes.io/projected/b05705b9-6c2f-4c7f-9d58-e832c56afde5-kube-api-access-x47mz\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:30.275725 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:30.275723    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b05705b9-6c2f-4c7f-9d58-e832c56afde5-maas-api-xks-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:30.354284 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:30.354254    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1faf30d6-dvscm"
Oct 31 23:23:30.354740 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:30.354285    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1faf30d6-dvscm" event={"ID":"b05705b9-6c2f-4c7f-9d58-e832c56afde5","Type":"ContainerDied","Data":"9d161b577b363c777ac19fbb1c8b7798ad60c2bdf72816cd91210c1be7ac70eb"}
Oct 31 23:23:30.354740 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:30.354352    4301 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d161b577b363c777ac19fbb1c8b7798ad60c2bdf72816cd91210c1be7ac70eb"
Oct 31 23:23:30.767520 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:30.765406    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1deb6-mmcrn"]
Oct 31 23:23:31.194414 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:31.194388    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:23:31.196700 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:31.196640    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:23:31.205496 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:31.205469    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:23:31.393917 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:31.393882    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1deb6-mmcrn" event={"ID":"7aa97ea3-555d-4b46-9b55-7e55c7481e0b","Type":"ContainerStarted","Data":"5c40bb0b084ae520954888258cc6f2d5d268c67a36d31741d38aa0004ae1ce0c"}
Oct 31 23:23:31.394717 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:31.394686    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1deb6-mmcrn" event={"ID":"7aa97ea3-555d-4b46-9b55-7e55c7481e0b","Type":"ContainerStarted","Data":"8118df4cf006acc89be003d2d1f0c7507cb66fe6f62dbf741e4a7adf0cbb4567"}
Oct 31 23:23:31.406975 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:31.406949    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3ef6611-0d56-41c3-9357-81e33a2fd67a" path="/var/lib/kubelet/pods/a3ef6611-0d56-41c3-9357-81e33a2fd67a/volumes"
Oct 31 23:23:31.414898 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:31.414854    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1deb6-mmcrn" podStartSLOduration=2.41484049 podStartE2EDuration="2.41484049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:23:31.410915446 +0000 UTC m=+2030.158104951" watchObservedRunningTime="2026-10-31 23:23:31.41484049 +0000 UTC m=+2030.162029995"
Oct 31 23:23:32.403285 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:32.403251    4301 generic.go:356] "Generic (PLEG): container finished" podID="7aa97ea3-555d-4b46-9b55-7e55c7481e0b" containerID="5c40bb0b084ae520954888258cc6f2d5d268c67a36d31741d38aa0004ae1ce0c" exitCode=0
Oct 31 23:23:32.403762 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:32.403337    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1deb6-mmcrn" event={"ID":"7aa97ea3-555d-4b46-9b55-7e55c7481e0b","Type":"ContainerDied","Data":"5c40bb0b084ae520954888258cc6f2d5d268c67a36d31741d38aa0004ae1ce0c"}
Oct 31 23:23:32.550504 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:23:32.550464    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:23:33.344675 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:33.344641    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6a08-j4xwt"]
Oct 31 23:23:33.345302 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:33.345288    4301 cpu_manager.go:414] "containerMap: removing container" podUID="b05705b9-6c2f-4c7f-9d58-e832c56afde5" containerName="revoke-keys"
Oct 31 23:23:33.345394 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:33.345303    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b05705b9-6c2f-4c7f-9d58-e832c56afde5" containerName="revoke-keys"
Oct 31 23:23:33.345445 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:33.345435    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="b05705b9-6c2f-4c7f-9d58-e832c56afde5" containerName="revoke-keys"
Oct 31 23:23:33.349058 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:33.349043    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6a08-j4xwt"
Oct 31 23:23:33.366739 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:33.366708    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6a08-j4xwt"]
Oct 31 23:23:33.464777 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:33.464735    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a44bea-af0f-499d-ae29-aa71d42e7853-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8e6a08-j4xwt\" (UID: \"b0a44bea-af0f-499d-ae29-aa71d42e7853\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6a08-j4xwt"
Oct 31 23:23:33.465214 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:33.464896    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a44bea-af0f-499d-ae29-aa71d42e7853-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8e6a08-j4xwt\" (UID: \"b0a44bea-af0f-499d-ae29-aa71d42e7853\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6a08-j4xwt"
Oct 31 23:23:33.465214 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:33.465041    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt8x8\" (UniqueName: \"kubernetes.io/projected/b0a44bea-af0f-499d-ae29-aa71d42e7853-kube-api-access-qt8x8\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8e6a08-j4xwt\" (UID: \"b0a44bea-af0f-499d-ae29-aa71d42e7853\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6a08-j4xwt"
Oct 31 23:23:33.566218 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:33.566185    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a44bea-af0f-499d-ae29-aa71d42e7853-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8e6a08-j4xwt\" (UID: \"b0a44bea-af0f-499d-ae29-aa71d42e7853\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6a08-j4xwt"
Oct 31 23:23:33.566372 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:33.566341    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt8x8\" (UniqueName: \"kubernetes.io/projected/b0a44bea-af0f-499d-ae29-aa71d42e7853-kube-api-access-qt8x8\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8e6a08-j4xwt\" (UID: \"b0a44bea-af0f-499d-ae29-aa71d42e7853\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6a08-j4xwt"
Oct 31 23:23:33.566495 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:33.566482    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a44bea-af0f-499d-ae29-aa71d42e7853-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8e6a08-j4xwt\" (UID: \"b0a44bea-af0f-499d-ae29-aa71d42e7853\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6a08-j4xwt"
Oct 31 23:23:33.566837 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:33.566817    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a44bea-af0f-499d-ae29-aa71d42e7853-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8e6a08-j4xwt\" (UID: \"b0a44bea-af0f-499d-ae29-aa71d42e7853\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6a08-j4xwt"
Oct 31 23:23:33.568784 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:33.568758    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a44bea-af0f-499d-ae29-aa71d42e7853-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8e6a08-j4xwt\" (UID: \"b0a44bea-af0f-499d-ae29-aa71d42e7853\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6a08-j4xwt"
Oct 31 23:23:33.577975 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:33.577956    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt8x8\" (UniqueName: \"kubernetes.io/projected/b0a44bea-af0f-499d-ae29-aa71d42e7853-kube-api-access-qt8x8\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8e6a08-j4xwt\" (UID: \"b0a44bea-af0f-499d-ae29-aa71d42e7853\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6a08-j4xwt"
Oct 31 23:23:33.602606 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:33.602547    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1deb6-mmcrn"
Oct 31 23:23:33.663866 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:33.663831    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6a08-j4xwt"
Oct 31 23:23:33.667262 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:33.667236    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7aa97ea3-555d-4b46-9b55-7e55c7481e0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa97ea3-555d-4b46-9b55-7e55c7481e0b-maas-api-xks-ca\") pod \"7aa97ea3-555d-4b46-9b55-7e55c7481e0b\" (UID: \"7aa97ea3-555d-4b46-9b55-7e55c7481e0b\") "
Oct 31 23:23:33.667487 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:33.667465    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aa97ea3-555d-4b46-9b55-7e55c7481e0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa97ea3-555d-4b46-9b55-7e55c7481e0b-maas-api-openshift-ca\") pod \"7aa97ea3-555d-4b46-9b55-7e55c7481e0b\" (UID: \"7aa97ea3-555d-4b46-9b55-7e55c7481e0b\") "
Oct 31 23:23:33.667600 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:33.667585    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aa97ea3-555d-4b46-9b55-7e55c7481e0b-kube-api-access-zbmvc\" (UniqueName: \"kubernetes.io/projected/7aa97ea3-555d-4b46-9b55-7e55c7481e0b-kube-api-access-zbmvc\") pod \"7aa97ea3-555d-4b46-9b55-7e55c7481e0b\" (UID: \"7aa97ea3-555d-4b46-9b55-7e55c7481e0b\") "
Oct 31 23:23:33.671200 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:33.670985    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa97ea3-555d-4b46-9b55-7e55c7481e0b-kube-api-access-zbmvc" pod "7aa97ea3-555d-4b46-9b55-7e55c7481e0b" (UID: "7aa97ea3-555d-4b46-9b55-7e55c7481e0b"). InnerVolumeSpecName "kube-api-access-zbmvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:23:33.671555 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:33.671498    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa97ea3-555d-4b46-9b55-7e55c7481e0b-maas-api-openshift-ca" pod "7aa97ea3-555d-4b46-9b55-7e55c7481e0b" (UID: "7aa97ea3-555d-4b46-9b55-7e55c7481e0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:23:33.677058 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:33.677029    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aa97ea3-555d-4b46-9b55-7e55c7481e0b-maas-api-xks-ca" pod "7aa97ea3-555d-4b46-9b55-7e55c7481e0b" (UID: "7aa97ea3-555d-4b46-9b55-7e55c7481e0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:23:33.770039 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:33.770008    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa97ea3-555d-4b46-9b55-7e55c7481e0b-maas-api-openshift-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:33.770039 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:33.770036    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbmvc\" (UniqueName: \"kubernetes.io/projected/7aa97ea3-555d-4b46-9b55-7e55c7481e0b-kube-api-access-zbmvc\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:33.770039 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:33.770046    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa97ea3-555d-4b46-9b55-7e55c7481e0b-maas-api-xks-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:34.041688 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:34.041651    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6a08-j4xwt"]
Oct 31 23:23:34.055408 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:23:34.053622    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a44bea_af0f_499d_ae29_aa71d42e7853.slice/crio-3643c170cfc438da1596441f3d8773894ae8197b9540d12ca5870ed791a364ec WatchSource:0}: Error finding container 3643c170cfc438da1596441f3d8773894ae8197b9540d12ca5870ed791a364ec: Status 404 returned error can't find the container with id 3643c170cfc438da1596441f3d8773894ae8197b9540d12ca5870ed791a364ec
Oct 31 23:23:34.421675 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:34.421639    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6a08-j4xwt" event={"ID":"b0a44bea-af0f-499d-ae29-aa71d42e7853","Type":"ContainerStarted","Data":"3643c170cfc438da1596441f3d8773894ae8197b9540d12ca5870ed791a364ec"}
Oct 31 23:23:34.423083 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:34.423055    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1deb6-mmcrn" event={"ID":"7aa97ea3-555d-4b46-9b55-7e55c7481e0b","Type":"ContainerDied","Data":"8118df4cf006acc89be003d2d1f0c7507cb66fe6f62dbf741e4a7adf0cbb4567"}
Oct 31 23:23:34.423083 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:34.423082    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1deb6-mmcrn"
Oct 31 23:23:34.423252 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:34.423102    4301 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8118df4cf006acc89be003d2d1f0c7507cb66fe6f62dbf741e4a7adf0cbb4567"
Oct 31 23:23:35.432897 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:35.432863    4301 generic.go:356] "Generic (PLEG): container finished" podID="b0a44bea-af0f-499d-ae29-aa71d42e7853" containerID="6f5795b70347257a963100a1f7ab094e8f4a12287a4938159d12cce0364e68db" exitCode=0
Oct 31 23:23:35.433461 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:35.432940    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6a08-j4xwt" event={"ID":"b0a44bea-af0f-499d-ae29-aa71d42e7853","Type":"ContainerDied","Data":"6f5795b70347257a963100a1f7ab094e8f4a12287a4938159d12cce0364e68db"}
Oct 31 23:23:36.620292 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:36.620261    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6a08-j4xwt"
Oct 31 23:23:36.762314 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:36.762223    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0a44bea-af0f-499d-ae29-aa71d42e7853-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a44bea-af0f-499d-ae29-aa71d42e7853-maas-api-xks-ca\") pod \"b0a44bea-af0f-499d-ae29-aa71d42e7853\" (UID: \"b0a44bea-af0f-499d-ae29-aa71d42e7853\") "
Oct 31 23:23:36.762530 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:36.762504    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0a44bea-af0f-499d-ae29-aa71d42e7853-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a44bea-af0f-499d-ae29-aa71d42e7853-maas-api-openshift-ca\") pod \"b0a44bea-af0f-499d-ae29-aa71d42e7853\" (UID: \"b0a44bea-af0f-499d-ae29-aa71d42e7853\") "
Oct 31 23:23:36.762712 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:36.762696    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0a44bea-af0f-499d-ae29-aa71d42e7853-kube-api-access-qt8x8\" (UniqueName: \"kubernetes.io/projected/b0a44bea-af0f-499d-ae29-aa71d42e7853-kube-api-access-qt8x8\") pod \"b0a44bea-af0f-499d-ae29-aa71d42e7853\" (UID: \"b0a44bea-af0f-499d-ae29-aa71d42e7853\") "
Oct 31 23:23:36.762859 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:36.762837    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0a44bea-af0f-499d-ae29-aa71d42e7853-maas-api-openshift-ca" pod "b0a44bea-af0f-499d-ae29-aa71d42e7853" (UID: "b0a44bea-af0f-499d-ae29-aa71d42e7853"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:23:36.763244 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:36.763227    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a44bea-af0f-499d-ae29-aa71d42e7853-maas-api-openshift-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:36.764393 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:36.764349    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0a44bea-af0f-499d-ae29-aa71d42e7853-maas-api-xks-ca" pod "b0a44bea-af0f-499d-ae29-aa71d42e7853" (UID: "b0a44bea-af0f-499d-ae29-aa71d42e7853"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:23:36.764741 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:36.764725    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a44bea-af0f-499d-ae29-aa71d42e7853-kube-api-access-qt8x8" pod "b0a44bea-af0f-499d-ae29-aa71d42e7853" (UID: "b0a44bea-af0f-499d-ae29-aa71d42e7853"). InnerVolumeSpecName "kube-api-access-qt8x8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:23:36.865227 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:36.865194    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt8x8\" (UniqueName: \"kubernetes.io/projected/b0a44bea-af0f-499d-ae29-aa71d42e7853-kube-api-access-qt8x8\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:36.865227 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:36.865225    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a44bea-af0f-499d-ae29-aa71d42e7853-maas-api-xks-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:36.992384 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:36.991888    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"]
Oct 31 23:23:37.015432 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.011046    4301 cpu_manager.go:414] "containerMap: removing container" podUID="7aa97ea3-555d-4b46-9b55-7e55c7481e0b" containerName="revoke-keys"
Oct 31 23:23:37.015432 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.011076    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aa97ea3-555d-4b46-9b55-7e55c7481e0b" containerName="revoke-keys"
Oct 31 23:23:37.015432 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.011124    4301 cpu_manager.go:414] "containerMap: removing container" podUID="b0a44bea-af0f-499d-ae29-aa71d42e7853" containerName="revoke-keys"
Oct 31 23:23:37.015432 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.011135    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0a44bea-af0f-499d-ae29-aa71d42e7853" containerName="revoke-keys"
Oct 31 23:23:37.015432 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.011314    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aa97ea3-555d-4b46-9b55-7e55c7481e0b" containerName="revoke-keys"
Oct 31 23:23:37.015432 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.011334    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0a44bea-af0f-499d-ae29-aa71d42e7853" containerName="revoke-keys"
Oct 31 23:23:37.041400 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.025257    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"]
Oct 31 23:23:37.041400 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.025402    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:23:37.193019 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.192987    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fd21e5d-47da-405b-9724-84cafcd106d2-istio-token\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-z8wck\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:23:37.193205 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.193065    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-istio-data\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-z8wck\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:23:37.193205 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.193175    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-credential-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-z8wck\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:23:37.193288 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.193259    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-istio-envoy\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-z8wck\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:23:37.193432 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.193407    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fd21e5d-47da-405b-9724-84cafcd106d2-istiod-ca-cert\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-z8wck\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:23:37.193514 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.193499    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fd21e5d-47da-405b-9724-84cafcd106d2-istio-podinfo\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-z8wck\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:23:37.193569 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.193550    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh9dl\" (UniqueName: \"kubernetes.io/projected/1fd21e5d-47da-405b-9724-84cafcd106d2-kube-api-access-zh9dl\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-z8wck\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:23:37.193689 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.193678    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-workload-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-z8wck\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:23:37.193725 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.193717    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-workload-certs\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-z8wck\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:23:37.294957 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.294875    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fd21e5d-47da-405b-9724-84cafcd106d2-istiod-ca-cert\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-z8wck\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:23:37.295108 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.294962    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fd21e5d-47da-405b-9724-84cafcd106d2-istio-podinfo\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-z8wck\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:23:37.295108 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.294995    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh9dl\" (UniqueName: \"kubernetes.io/projected/1fd21e5d-47da-405b-9724-84cafcd106d2-kube-api-access-zh9dl\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-z8wck\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:23:37.295108 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.295088    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-workload-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-z8wck\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:23:37.295214 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.295134    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-workload-certs\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-z8wck\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:23:37.295214 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.295174    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fd21e5d-47da-405b-9724-84cafcd106d2-istio-token\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-z8wck\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:23:37.295285 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.295258    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-istio-data\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-z8wck\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:23:37.295345 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.295289    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-credential-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-z8wck\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:23:37.295480 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.295396    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-istio-envoy\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-z8wck\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:23:37.295652 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.295624    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-workload-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-z8wck\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:23:37.295757 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.295686    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-workload-certs\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-z8wck\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:23:37.295757 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.295723    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-istio-data\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-z8wck\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:23:37.295877 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.295791    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fd21e5d-47da-405b-9724-84cafcd106d2-istiod-ca-cert\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-z8wck\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:23:37.295877 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.295861    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-credential-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-z8wck\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:23:37.297709 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.297689    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fd21e5d-47da-405b-9724-84cafcd106d2-istio-podinfo\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-z8wck\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:23:37.297911 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.297894    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-istio-envoy\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-z8wck\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:23:37.306711 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.306689    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fd21e5d-47da-405b-9724-84cafcd106d2-istio-token\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-z8wck\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:23:37.306895 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.306882    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh9dl\" (UniqueName: \"kubernetes.io/projected/1fd21e5d-47da-405b-9724-84cafcd106d2-kube-api-access-zh9dl\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-z8wck\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:23:37.391849 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.391815    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:23:37.452881 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.452853    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6a08-j4xwt"
Oct 31 23:23:37.452988 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.452935    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6a08-j4xwt" event={"ID":"b0a44bea-af0f-499d-ae29-aa71d42e7853","Type":"ContainerDied","Data":"3643c170cfc438da1596441f3d8773894ae8197b9540d12ca5870ed791a364ec"}
Oct 31 23:23:37.453062 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.452987    4301 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3643c170cfc438da1596441f3d8773894ae8197b9540d12ca5870ed791a364ec"
Oct 31 23:23:37.739680 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.739643    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"]
Oct 31 23:23:37.744864 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:23:37.744836    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fd21e5d_47da_405b_9724_84cafcd106d2.slice/crio-80a7e196fb1e99222e6ed04809e96aee6f8573d5d94b8802fbdae6390319faca WatchSource:0}: Error finding container 80a7e196fb1e99222e6ed04809e96aee6f8573d5d94b8802fbdae6390319faca: Status 404 returned error can't find the container with id 80a7e196fb1e99222e6ed04809e96aee6f8573d5d94b8802fbdae6390319faca
Oct 31 23:23:37.748044 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.748007    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:23:37.748142 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.748081    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:23:37.748142 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:37.748110    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:23:38.462972 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:38.462937    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck" event={"ID":"1fd21e5d-47da-405b-9724-84cafcd106d2","Type":"ContainerStarted","Data":"dcbecb35bcaa54742a40464e3698a8b9203ca9a0d259d6cc801813a77e8aec74"}
Oct 31 23:23:38.463156 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:38.462992    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck" event={"ID":"1fd21e5d-47da-405b-9724-84cafcd106d2","Type":"ContainerStarted","Data":"80a7e196fb1e99222e6ed04809e96aee6f8573d5d94b8802fbdae6390319faca"}
Oct 31 23:23:38.552388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:38.547598    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck" podStartSLOduration=2.547574482 podStartE2EDuration="2.547574482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:23:38.488926823 +0000 UTC m=+2037.236116330" watchObservedRunningTime="2026-10-31 23:23:38.547574482 +0000 UTC m=+2037.294763984"
Oct 31 23:23:39.395855 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:39.395763    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:23:39.403531 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:39.403495    4301 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 31 23:23:39.403717 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:39.403557    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck" podUID="1fd21e5d-47da-405b-9724-84cafcd106d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 31 23:23:40.394847 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:40.394809    4301 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 31 23:23:40.395196 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:40.395168    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck" podUID="1fd21e5d-47da-405b-9724-84cafcd106d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 31 23:23:41.392799 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:41.392770    4301 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 31 23:23:41.393255 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:41.393231    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck" podUID="1fd21e5d-47da-405b-9724-84cafcd106d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 31 23:23:42.138382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:42.120872    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4eff5f-2qsml"]
Oct 31 23:23:42.214389 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:42.198330    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4eff5f-2qsml"]
Oct 31 23:23:42.417581 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:42.417477    4301 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 31 23:23:42.418489 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:42.418457    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck" podUID="1fd21e5d-47da-405b-9724-84cafcd106d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 31 23:23:43.393919 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:43.393879    4301 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 31 23:23:43.394194 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:43.394168    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck" podUID="1fd21e5d-47da-405b-9724-84cafcd106d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 31 23:23:43.445037 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:43.444988    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af80a837-3fb4-4f97-bd01-0268b934397f" path="/var/lib/kubelet/pods/af80a837-3fb4-4f97-bd01-0268b934397f/volumes"
Oct 31 23:23:44.393776 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:44.393258    4301 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 31 23:23:44.393776 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:44.393325    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck" podUID="1fd21e5d-47da-405b-9724-84cafcd106d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 31 23:23:45.177472 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:45.177429    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-885fdf-nxpx8"]
Oct 31 23:23:45.253771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:45.253716    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-885fdf-nxpx8"]
Oct 31 23:23:45.406198 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:45.406165    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34d4f0d4-70aa-4c54-8a10-82677b2af2f2" path="/var/lib/kubelet/pods/34d4f0d4-70aa-4c54-8a10-82677b2af2f2/volumes"
Oct 31 23:23:45.452817 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:45.441814    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:23:45.452817 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:45.443731    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:23:45.455501 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:45.455472    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:23:49.723090 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:49.723063    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"]
Oct 31 23:23:49.723657 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:49.723626    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62" podUID="d05b75f2-18af-4d86-99d9-a51d2d23c3b5" containerName="istio-proxy" containerID="cri-o://920f66b2cd028172d5b338559802c2525925afcdb8927e7363df4d657c4a1107" gracePeriod=30
Oct 31 23:23:50.039751 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:50.039640    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:23:52.168543 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:52.168488    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:52.383515 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:52.383464    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:52.430268 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:52.429567    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:52.515273 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:52.514465    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 23:23:52.527502 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:52.517967    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:52.527502 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:52.523951    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:52.557650 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:52.557594    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 23:23:52.599476 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:52.598468    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 23:23:52.651373 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:52.651314    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:23:53.902112 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:53.902076    4301 patch_prober.go:29] interesting pod/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 23:23:53.903119 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:53.903085    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62" podUID="d05b75f2-18af-4d86-99d9-a51d2d23c3b5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 23:23:53.904326 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:53.904255    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"]
Oct 31 23:23:53.904620 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:53.904538    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz" podUID="e5c2b8fb-331d-4df2-ad30-13549d8a660c" containerName="istio-proxy" containerID="cri-o://222190fd321ae01f361feeaa7c2fe1704ccc18b1be65e3f88b25bbb3ccc70e1f" gracePeriod=30
Oct 31 23:23:53.924065 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:53.923195    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"]
Oct 31 23:23:53.924065 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:53.923446    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7" podUID="46881476-3103-4ee9-9b90-ec0ba9f30c3e" containerName="istio-proxy" containerID="cri-o://f27c427c52e9908a305179ea8354e0690e33ac07fa37c31b64fe479465bd5033" gracePeriod=30
Oct 31 23:23:53.971495 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:53.971457    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"]
Oct 31 23:23:53.971721 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:53.971696    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh" podUID="b1a1157d-88ea-448f-931d-9528c5df5684" containerName="istio-proxy" containerID="cri-o://fa51886660aad72a3e8fab6906996840346df64773a5627edb1abfd47c6a2609" gracePeriod=30
Oct 31 23:23:54.479543 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:54.479510    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2126ff-7gqc9"]
Oct 31 23:23:54.485138 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:54.485114    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2126ff-7gqc9"
Oct 31 23:23:54.497683 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:54.497657    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 23:23:54.497823 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:54.497714    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\""
Oct 31 23:23:54.498713 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:54.498693    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2126ff-7gqc9"]
Oct 31 23:23:54.590960 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:54.590921    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a0fbeac-3a35-47cc-89fe-dcd510841e18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2126ff-7gqc9\" (UID: \"0a0fbeac-3a35-47cc-89fe-dcd510841e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2126ff-7gqc9"
Oct 31 23:23:54.591143 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:54.591072    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a0fbeac-3a35-47cc-89fe-dcd510841e18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2126ff-7gqc9\" (UID: \"0a0fbeac-3a35-47cc-89fe-dcd510841e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2126ff-7gqc9"
Oct 31 23:23:54.591200 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:54.591138    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t5xn\" (UniqueName: \"kubernetes.io/projected/0a0fbeac-3a35-47cc-89fe-dcd510841e18-kube-api-access-9t5xn\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2126ff-7gqc9\" (UID: \"0a0fbeac-3a35-47cc-89fe-dcd510841e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2126ff-7gqc9"
Oct 31 23:23:54.692947 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:54.692908    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a0fbeac-3a35-47cc-89fe-dcd510841e18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2126ff-7gqc9\" (UID: \"0a0fbeac-3a35-47cc-89fe-dcd510841e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2126ff-7gqc9"
Oct 31 23:23:54.693129 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:54.693012    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a0fbeac-3a35-47cc-89fe-dcd510841e18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2126ff-7gqc9\" (UID: \"0a0fbeac-3a35-47cc-89fe-dcd510841e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2126ff-7gqc9"
Oct 31 23:23:54.693129 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:54.693087    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t5xn\" (UniqueName: \"kubernetes.io/projected/0a0fbeac-3a35-47cc-89fe-dcd510841e18-kube-api-access-9t5xn\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2126ff-7gqc9\" (UID: \"0a0fbeac-3a35-47cc-89fe-dcd510841e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2126ff-7gqc9"
Oct 31 23:23:54.693681 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:54.693662    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a0fbeac-3a35-47cc-89fe-dcd510841e18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2126ff-7gqc9\" (UID: \"0a0fbeac-3a35-47cc-89fe-dcd510841e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2126ff-7gqc9"
Oct 31 23:23:54.695416 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:54.695397    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a0fbeac-3a35-47cc-89fe-dcd510841e18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2126ff-7gqc9\" (UID: \"0a0fbeac-3a35-47cc-89fe-dcd510841e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2126ff-7gqc9"
Oct 31 23:23:54.705808 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:54.705771    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t5xn\" (UniqueName: \"kubernetes.io/projected/0a0fbeac-3a35-47cc-89fe-dcd510841e18-kube-api-access-9t5xn\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2126ff-7gqc9\" (UID: \"0a0fbeac-3a35-47cc-89fe-dcd510841e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2126ff-7gqc9"
Oct 31 23:23:54.806191 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:54.806161    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2126ff-7gqc9"
Oct 31 23:23:54.827273 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:54.827196    4301 generic.go:356] "Generic (PLEG): container finished" podID="d05b75f2-18af-4d86-99d9-a51d2d23c3b5" containerID="920f66b2cd028172d5b338559802c2525925afcdb8927e7363df4d657c4a1107" exitCode=0
Oct 31 23:23:54.827415 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:54.827278    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62" event={"ID":"d05b75f2-18af-4d86-99d9-a51d2d23c3b5","Type":"ContainerDied","Data":"920f66b2cd028172d5b338559802c2525925afcdb8927e7363df4d657c4a1107"}
Oct 31 23:23:54.827415 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:54.827394    4301 scope.go:122] "RemoveContainer" containerID="4a30b386bc535202ee7f79b1b6b7bbceaac019d88ca04099ae4c0367bf107eba"
Oct 31 23:23:55.119371 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.119326    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:23:55.219612 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.219575    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istio-token\" (UniqueName: \"kubernetes.io/projected/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istio-token\") pod \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") "
Oct 31 23:23:55.219798 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.219636    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istio-envoy\") pod \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") "
Oct 31 23:23:55.219798 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.219671    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istiod-ca-cert\") pod \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") "
Oct 31 23:23:55.219798 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.219764    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istio-data\") pod \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") "
Oct 31 23:23:55.219798 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.219794    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-kube-api-access-4pvwq\" (UniqueName: \"kubernetes.io/projected/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-kube-api-access-4pvwq\") pod \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") "
Oct 31 23:23:55.220053 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.219908    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-credential-socket\") pod \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") "
Oct 31 23:23:55.220053 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.219998    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-workload-socket\") pod \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") "
Oct 31 23:23:55.220161 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.220058    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istio-podinfo\") pod \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") "
Oct 31 23:23:55.220161 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.220085    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-workload-certs\") pod \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\" (UID: \"d05b75f2-18af-4d86-99d9-a51d2d23c3b5\") "
Oct 31 23:23:55.220721 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.220689    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istiod-ca-cert" pod "d05b75f2-18af-4d86-99d9-a51d2d23c3b5" (UID: "d05b75f2-18af-4d86-99d9-a51d2d23c3b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:23:55.220936 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.220910    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-workload-certs" pod "d05b75f2-18af-4d86-99d9-a51d2d23c3b5" (UID: "d05b75f2-18af-4d86-99d9-a51d2d23c3b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:55.221045 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.221020    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:55.221118 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.221103    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-credential-socket" pod "d05b75f2-18af-4d86-99d9-a51d2d23c3b5" (UID: "d05b75f2-18af-4d86-99d9-a51d2d23c3b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:55.221338 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.221315    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-workload-socket" pod "d05b75f2-18af-4d86-99d9-a51d2d23c3b5" (UID: "d05b75f2-18af-4d86-99d9-a51d2d23c3b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:55.221599 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.221559    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istio-data" pod "d05b75f2-18af-4d86-99d9-a51d2d23c3b5" (UID: "d05b75f2-18af-4d86-99d9-a51d2d23c3b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:55.222970 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.222936    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istio-envoy" pod "d05b75f2-18af-4d86-99d9-a51d2d23c3b5" (UID: "d05b75f2-18af-4d86-99d9-a51d2d23c3b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:55.223190 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.223171    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-kube-api-access-4pvwq" pod "d05b75f2-18af-4d86-99d9-a51d2d23c3b5" (UID: "d05b75f2-18af-4d86-99d9-a51d2d23c3b5"). InnerVolumeSpecName "kube-api-access-4pvwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:23:55.223279 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.223197    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istio-token" pod "d05b75f2-18af-4d86-99d9-a51d2d23c3b5" (UID: "d05b75f2-18af-4d86-99d9-a51d2d23c3b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:23:55.223542 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.223516    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istio-podinfo" pod "d05b75f2-18af-4d86-99d9-a51d2d23c3b5" (UID: "d05b75f2-18af-4d86-99d9-a51d2d23c3b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:23:55.248643 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:23:55.248582    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a0fbeac_3a35_47cc_89fe_dcd510841e18.slice/crio-67e090f48a8bff669f9ba3debbd8281de9e7805f5ea01f19951e5995316ee770 WatchSource:0}: Error finding container 67e090f48a8bff669f9ba3debbd8281de9e7805f5ea01f19951e5995316ee770: Status 404 returned error can't find the container with id 67e090f48a8bff669f9ba3debbd8281de9e7805f5ea01f19951e5995316ee770
Oct 31 23:23:55.262417 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.262378    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2126ff-7gqc9"]
Oct 31 23:23:55.322327 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.322303    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:55.322327 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.322327    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:55.322510 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.322344    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:55.322510 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.322365    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:55.322510 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.322373    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:55.322510 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.322382    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:55.322510 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.322390    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pvwq\" (UniqueName: \"kubernetes.io/projected/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-kube-api-access-4pvwq\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:55.322510 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.322401    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05b75f2-18af-4d86-99d9-a51d2d23c3b5-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:55.827172 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.827134    4301 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 23:23:55.827337 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.827186    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh" podUID="b1a1157d-88ea-448f-931d-9528c5df5684" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 23:23:55.860010 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.859977    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"
Oct 31 23:23:55.860010 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.859985    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62" event={"ID":"d05b75f2-18af-4d86-99d9-a51d2d23c3b5","Type":"ContainerDied","Data":"0833dac9bbe70569d8f85a4a7028af2480f46a2526e0f0977c1392cb6be07926"}
Oct 31 23:23:55.860212 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.860057    4301 scope.go:122] "RemoveContainer" containerID="920f66b2cd028172d5b338559802c2525925afcdb8927e7363df4d657c4a1107"
Oct 31 23:23:55.862206 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.862178    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2126ff-7gqc9" event={"ID":"0a0fbeac-3a35-47cc-89fe-dcd510841e18","Type":"ContainerStarted","Data":"b49bd1b04c356737e52b6b8109a85a709ad6f60cd47b09da1fc5ab81f8b444b4"}
Oct 31 23:23:55.862317 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.862266    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2126ff-7gqc9" event={"ID":"0a0fbeac-3a35-47cc-89fe-dcd510841e18","Type":"ContainerStarted","Data":"67e090f48a8bff669f9ba3debbd8281de9e7805f5ea01f19951e5995316ee770"}
Oct 31 23:23:55.879990 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.879967    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"]
Oct 31 23:23:55.885570 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.885545    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-m6g62"]
Oct 31 23:23:55.899082 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:55.899035    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2126ff-7gqc9" podStartSLOduration=1.899022577 podStartE2EDuration="1.899022577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:23:55.895193937 +0000 UTC m=+2054.642383447" watchObservedRunningTime="2026-10-31 23:23:55.899022577 +0000 UTC m=+2054.646212081"
Oct 31 23:23:56.054018 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:56.053885    4301 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 23:23:56.054263 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:56.054237    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz" podUID="e5c2b8fb-331d-4df2-ad30-13549d8a660c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 23:23:56.878154 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:56.878114    4301 generic.go:356] "Generic (PLEG): container finished" podID="0a0fbeac-3a35-47cc-89fe-dcd510841e18" containerID="b49bd1b04c356737e52b6b8109a85a709ad6f60cd47b09da1fc5ab81f8b444b4" exitCode=0
Oct 31 23:23:56.878697 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:56.878244    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2126ff-7gqc9" event={"ID":"0a0fbeac-3a35-47cc-89fe-dcd510841e18","Type":"ContainerDied","Data":"b49bd1b04c356737e52b6b8109a85a709ad6f60cd47b09da1fc5ab81f8b444b4"}
Oct 31 23:23:56.948388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:56.947425    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6a08-j4xwt"]
Oct 31 23:23:56.970427 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:56.970258    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6a08-j4xwt"]
Oct 31 23:23:57.685420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:57.671772    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0a44bea-af0f-499d-ae29-aa71d42e7853" path="/var/lib/kubelet/pods/b0a44bea-af0f-499d-ae29-aa71d42e7853/volumes"
Oct 31 23:23:57.685420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:57.672576    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d05b75f2-18af-4d86-99d9-a51d2d23c3b5" path="/var/lib/kubelet/pods/d05b75f2-18af-4d86-99d9-a51d2d23c3b5/volumes"
Oct 31 23:23:57.685420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:57.673026    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba9088-nclfq"]
Oct 31 23:23:57.717385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:57.716141    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba9088-nclfq"]
Oct 31 23:23:58.090797 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:58.090772    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2126ff-7gqc9"
Oct 31 23:23:58.222227 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:58.222137    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a0fbeac-3a35-47cc-89fe-dcd510841e18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a0fbeac-3a35-47cc-89fe-dcd510841e18-maas-api-xks-ca\") pod \"0a0fbeac-3a35-47cc-89fe-dcd510841e18\" (UID: \"0a0fbeac-3a35-47cc-89fe-dcd510841e18\") "
Oct 31 23:23:58.222404 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:58.222284    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a0fbeac-3a35-47cc-89fe-dcd510841e18-kube-api-access-9t5xn\" (UniqueName: \"kubernetes.io/projected/0a0fbeac-3a35-47cc-89fe-dcd510841e18-kube-api-access-9t5xn\") pod \"0a0fbeac-3a35-47cc-89fe-dcd510841e18\" (UID: \"0a0fbeac-3a35-47cc-89fe-dcd510841e18\") "
Oct 31 23:23:58.222572 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:58.222552    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a0fbeac-3a35-47cc-89fe-dcd510841e18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a0fbeac-3a35-47cc-89fe-dcd510841e18-maas-api-openshift-ca\") pod \"0a0fbeac-3a35-47cc-89fe-dcd510841e18\" (UID: \"0a0fbeac-3a35-47cc-89fe-dcd510841e18\") "
Oct 31 23:23:58.222942 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:58.222912    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a0fbeac-3a35-47cc-89fe-dcd510841e18-maas-api-openshift-ca" pod "0a0fbeac-3a35-47cc-89fe-dcd510841e18" (UID: "0a0fbeac-3a35-47cc-89fe-dcd510841e18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:23:58.223185 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:58.223166    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a0fbeac-3a35-47cc-89fe-dcd510841e18-maas-api-openshift-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:58.224303 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:58.224279    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a0fbeac-3a35-47cc-89fe-dcd510841e18-maas-api-xks-ca" pod "0a0fbeac-3a35-47cc-89fe-dcd510841e18" (UID: "0a0fbeac-3a35-47cc-89fe-dcd510841e18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:23:58.224470 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:58.224421    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a0fbeac-3a35-47cc-89fe-dcd510841e18-kube-api-access-9t5xn" pod "0a0fbeac-3a35-47cc-89fe-dcd510841e18" (UID: "0a0fbeac-3a35-47cc-89fe-dcd510841e18"). InnerVolumeSpecName "kube-api-access-9t5xn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:23:58.340595 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:58.337671    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a0fbeac-3a35-47cc-89fe-dcd510841e18-maas-api-xks-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:58.340595 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:58.337744    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t5xn\" (UniqueName: \"kubernetes.io/projected/0a0fbeac-3a35-47cc-89fe-dcd510841e18-kube-api-access-9t5xn\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:59.062973 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.062938    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2126ff-7gqc9" event={"ID":"0a0fbeac-3a35-47cc-89fe-dcd510841e18","Type":"ContainerDied","Data":"67e090f48a8bff669f9ba3debbd8281de9e7805f5ea01f19951e5995316ee770"}
Oct 31 23:23:59.063229 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.063187    4301 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67e090f48a8bff669f9ba3debbd8281de9e7805f5ea01f19951e5995316ee770"
Oct 31 23:23:59.063461 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.063446    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2126ff-7gqc9"
Oct 31 23:23:59.309647 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.309602    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:23:59.314123 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.314094    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:23:59.376605 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.376571    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:23:59.410288 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.410252    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdcd9409-f9de-4955-ab65-5a032181d22c" path="/var/lib/kubelet/pods/cdcd9409-f9de-4955-ab65-5a032181d22c/volumes"
Oct 31 23:23:59.490314 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.490287    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1a1157d-88ea-448f-931d-9528c5df5684-istio-token\" (UniqueName: \"kubernetes.io/projected/b1a1157d-88ea-448f-931d-9528c5df5684-istio-token\") pod \"b1a1157d-88ea-448f-931d-9528c5df5684\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") "
Oct 31 23:23:59.490523 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.490327    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-workload-certs\") pod \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") "
Oct 31 23:23:59.490523 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.490392    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-credential-socket\") pod \"b1a1157d-88ea-448f-931d-9528c5df5684\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") "
Oct 31 23:23:59.490523 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.490440    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istio-envoy\") pod \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") "
Oct 31 23:23:59.490523 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.490478    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1a1157d-88ea-448f-931d-9528c5df5684-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1a1157d-88ea-448f-931d-9528c5df5684-istiod-ca-cert\") pod \"b1a1157d-88ea-448f-931d-9528c5df5684\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") "
Oct 31 23:23:59.490523 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.490496    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-credential-socket\") pod \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") "
Oct 31 23:23:59.490804 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.490529    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-workload-socket\") pod \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") "
Oct 31 23:23:59.490804 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.490573    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istio-podinfo\") pod \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") "
Oct 31 23:23:59.490804 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.490603    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1a1157d-88ea-448f-931d-9528c5df5684-kube-api-access-zbxrn\" (UniqueName: \"kubernetes.io/projected/b1a1157d-88ea-448f-931d-9528c5df5684-kube-api-access-zbxrn\") pod \"b1a1157d-88ea-448f-931d-9528c5df5684\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") "
Oct 31 23:23:59.490804 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.490655    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5c2b8fb-331d-4df2-ad30-13549d8a660c-kube-api-access-2vgjv\" (UniqueName: \"kubernetes.io/projected/e5c2b8fb-331d-4df2-ad30-13549d8a660c-kube-api-access-2vgjv\") pod \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") "
Oct 31 23:23:59.490804 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.490685    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istio-token\" (UniqueName: \"kubernetes.io/projected/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istio-token\") pod \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") "
Oct 31 23:23:59.490804 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.490699    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-credential-socket" pod "b1a1157d-88ea-448f-931d-9528c5df5684" (UID: "b1a1157d-88ea-448f-931d-9528c5df5684"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:59.490804 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.490715    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istio-podinfo\") pod \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") "
Oct 31 23:23:59.491162 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.490844    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istio-data\") pod \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") "
Oct 31 23:23:59.491162 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.490889    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-workload-certs\") pod \"b1a1157d-88ea-448f-931d-9528c5df5684\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") "
Oct 31 23:23:59.491162 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.490960    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46881476-3103-4ee9-9b90-ec0ba9f30c3e-kube-api-access-ctk55\" (UniqueName: \"kubernetes.io/projected/46881476-3103-4ee9-9b90-ec0ba9f30c3e-kube-api-access-ctk55\") pod \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") "
Oct 31 23:23:59.491162 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.490988    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istio-token\" (UniqueName: \"kubernetes.io/projected/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istio-token\") pod \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") "
Oct 31 23:23:59.491162 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.491054    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-workload-socket\") pod \"b1a1157d-88ea-448f-931d-9528c5df5684\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") "
Oct 31 23:23:59.491162 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.491117    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-workload-certs\") pod \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") "
Oct 31 23:23:59.491877 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.491160    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-workload-socket" pod "e5c2b8fb-331d-4df2-ad30-13549d8a660c" (UID: "e5c2b8fb-331d-4df2-ad30-13549d8a660c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:59.491877 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.491228    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istio-envoy\") pod \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") "
Oct 31 23:23:59.491877 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.491303    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1a1157d-88ea-448f-931d-9528c5df5684-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1a1157d-88ea-448f-931d-9528c5df5684-istio-podinfo\") pod \"b1a1157d-88ea-448f-931d-9528c5df5684\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") "
Oct 31 23:23:59.491877 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.491400    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-credential-socket\") pod \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") "
Oct 31 23:23:59.491877 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.491444    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istio-data\") pod \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") "
Oct 31 23:23:59.491877 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.491483    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-istio-envoy\") pod \"b1a1157d-88ea-448f-931d-9528c5df5684\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") "
Oct 31 23:23:59.491877 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.491501    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1a1157d-88ea-448f-931d-9528c5df5684-istiod-ca-cert" pod "b1a1157d-88ea-448f-931d-9528c5df5684" (UID: "b1a1157d-88ea-448f-931d-9528c5df5684"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:23:59.491877 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.491583    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-istio-data\") pod \"b1a1157d-88ea-448f-931d-9528c5df5684\" (UID: \"b1a1157d-88ea-448f-931d-9528c5df5684\") "
Oct 31 23:23:59.491877 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.491617    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istiod-ca-cert\") pod \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\" (UID: \"e5c2b8fb-331d-4df2-ad30-13549d8a660c\") "
Oct 31 23:23:59.491877 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.491665    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-workload-socket\") pod \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") "
Oct 31 23:23:59.491877 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.491715    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istiod-ca-cert\") pod \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\" (UID: \"46881476-3103-4ee9-9b90-ec0ba9f30c3e\") "
Oct 31 23:23:59.491877 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.491792    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-credential-socket" pod "46881476-3103-4ee9-9b90-ec0ba9f30c3e" (UID: "46881476-3103-4ee9-9b90-ec0ba9f30c3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:59.492472 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.491918    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-workload-certs" pod "46881476-3103-4ee9-9b90-ec0ba9f30c3e" (UID: "46881476-3103-4ee9-9b90-ec0ba9f30c3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:59.492472 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.492122    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-workload-certs" pod "e5c2b8fb-331d-4df2-ad30-13549d8a660c" (UID: "e5c2b8fb-331d-4df2-ad30-13549d8a660c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:59.492472 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.492300    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-workload-socket" pod "b1a1157d-88ea-448f-931d-9528c5df5684" (UID: "b1a1157d-88ea-448f-931d-9528c5df5684"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:59.492686 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.492664    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-workload-socket" pod "46881476-3103-4ee9-9b90-ec0ba9f30c3e" (UID: "46881476-3103-4ee9-9b90-ec0ba9f30c3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:59.493166 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.493141    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1a1157d-88ea-448f-931d-9528c5df5684-istio-token" pod "b1a1157d-88ea-448f-931d-9528c5df5684" (UID: "b1a1157d-88ea-448f-931d-9528c5df5684"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:23:59.493310 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.493291    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1a1157d-88ea-448f-931d-9528c5df5684-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:59.493420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.493317    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:59.493420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.493333    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:59.493420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.493334    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-istio-data" pod "b1a1157d-88ea-448f-931d-9528c5df5684" (UID: "b1a1157d-88ea-448f-931d-9528c5df5684"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:59.493420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.493349    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1a1157d-88ea-448f-931d-9528c5df5684-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:59.493420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.493379    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:59.493420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.493393    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:59.493420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.493409    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:59.493808 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.493424    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:59.493808 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.493439    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:59.493808 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.493557    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-credential-socket" pod "e5c2b8fb-331d-4df2-ad30-13549d8a660c" (UID: "e5c2b8fb-331d-4df2-ad30-13549d8a660c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:59.493808 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.493677    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istiod-ca-cert" pod "e5c2b8fb-331d-4df2-ad30-13549d8a660c" (UID: "e5c2b8fb-331d-4df2-ad30-13549d8a660c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:23:59.493808 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.493739    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istiod-ca-cert" pod "46881476-3103-4ee9-9b90-ec0ba9f30c3e" (UID: "46881476-3103-4ee9-9b90-ec0ba9f30c3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:23:59.494064 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.493823    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istio-data" pod "e5c2b8fb-331d-4df2-ad30-13549d8a660c" (UID: "e5c2b8fb-331d-4df2-ad30-13549d8a660c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:59.494064 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.494021    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-workload-certs" pod "b1a1157d-88ea-448f-931d-9528c5df5684" (UID: "b1a1157d-88ea-448f-931d-9528c5df5684"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:59.494064 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.494043    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istio-data" pod "46881476-3103-4ee9-9b90-ec0ba9f30c3e" (UID: "46881476-3103-4ee9-9b90-ec0ba9f30c3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:59.495557 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.495531    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istio-envoy" pod "46881476-3103-4ee9-9b90-ec0ba9f30c3e" (UID: "46881476-3103-4ee9-9b90-ec0ba9f30c3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:59.495675 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.495581    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istio-token" pod "e5c2b8fb-331d-4df2-ad30-13549d8a660c" (UID: "e5c2b8fb-331d-4df2-ad30-13549d8a660c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:23:59.495895 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.495848    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istio-podinfo" pod "e5c2b8fb-331d-4df2-ad30-13549d8a660c" (UID: "e5c2b8fb-331d-4df2-ad30-13549d8a660c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:23:59.495895 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.495866    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istio-envoy" pod "e5c2b8fb-331d-4df2-ad30-13549d8a660c" (UID: "e5c2b8fb-331d-4df2-ad30-13549d8a660c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:59.495895 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.495864    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istio-podinfo" pod "46881476-3103-4ee9-9b90-ec0ba9f30c3e" (UID: "46881476-3103-4ee9-9b90-ec0ba9f30c3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:23:59.496282 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.496259    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-istio-envoy" pod "b1a1157d-88ea-448f-931d-9528c5df5684" (UID: "b1a1157d-88ea-448f-931d-9528c5df5684"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:23:59.496661 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.496641    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istio-token" pod "46881476-3103-4ee9-9b90-ec0ba9f30c3e" (UID: "46881476-3103-4ee9-9b90-ec0ba9f30c3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:23:59.496733 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.496670    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46881476-3103-4ee9-9b90-ec0ba9f30c3e-kube-api-access-ctk55" pod "46881476-3103-4ee9-9b90-ec0ba9f30c3e" (UID: "46881476-3103-4ee9-9b90-ec0ba9f30c3e"). InnerVolumeSpecName "kube-api-access-ctk55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:23:59.497609 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.497564    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c2b8fb-331d-4df2-ad30-13549d8a660c-kube-api-access-2vgjv" pod "e5c2b8fb-331d-4df2-ad30-13549d8a660c" (UID: "e5c2b8fb-331d-4df2-ad30-13549d8a660c"). InnerVolumeSpecName "kube-api-access-2vgjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:23:59.497752 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.497732    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1a1157d-88ea-448f-931d-9528c5df5684-kube-api-access-zbxrn" pod "b1a1157d-88ea-448f-931d-9528c5df5684" (UID: "b1a1157d-88ea-448f-931d-9528c5df5684"). InnerVolumeSpecName "kube-api-access-zbxrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:23:59.497796 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.497784    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1a1157d-88ea-448f-931d-9528c5df5684-istio-podinfo" pod "b1a1157d-88ea-448f-931d-9528c5df5684" (UID: "b1a1157d-88ea-448f-931d-9528c5df5684"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:23:59.595157 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.595121    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:59.595157 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.595154    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:59.595157 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.595164    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbxrn\" (UniqueName: \"kubernetes.io/projected/b1a1157d-88ea-448f-931d-9528c5df5684-kube-api-access-zbxrn\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:59.595417 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.595174    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vgjv\" (UniqueName: \"kubernetes.io/projected/e5c2b8fb-331d-4df2-ad30-13549d8a660c-kube-api-access-2vgjv\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:59.595417 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.595184    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:59.595417 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.595192    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:59.595417 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.595201    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:59.595417 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.595209    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:59.595417 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.595218    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctk55\" (UniqueName: \"kubernetes.io/projected/46881476-3103-4ee9-9b90-ec0ba9f30c3e-kube-api-access-ctk55\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:59.595417 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.595225    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:59.595417 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.595233    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:59.595417 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.595242    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1a1157d-88ea-448f-931d-9528c5df5684-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:59.595417 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.595250    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2b8fb-331d-4df2-ad30-13549d8a660c-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:59.595417 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.595258    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:59.595417 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.595266    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:59.595417 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.595274    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1a1157d-88ea-448f-931d-9528c5df5684-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:59.595417 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.595282    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c2b8fb-331d-4df2-ad30-13549d8a660c-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:23:59.595417 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:23:59.595291    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46881476-3103-4ee9-9b90-ec0ba9f30c3e-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:00.074773 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:00.074736    4301 generic.go:356] "Generic (PLEG): container finished" podID="b1a1157d-88ea-448f-931d-9528c5df5684" containerID="fa51886660aad72a3e8fab6906996840346df64773a5627edb1abfd47c6a2609" exitCode=0
Oct 31 23:24:00.074773 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:00.074781    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"
Oct 31 23:24:00.075087 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:00.074816    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh" event={"ID":"b1a1157d-88ea-448f-931d-9528c5df5684","Type":"ContainerDied","Data":"fa51886660aad72a3e8fab6906996840346df64773a5627edb1abfd47c6a2609"}
Oct 31 23:24:00.075087 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:00.074872    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh" event={"ID":"b1a1157d-88ea-448f-931d-9528c5df5684","Type":"ContainerDied","Data":"5e9ad06e2f7a735235737837f7071704e0a9eafa0b6ec9f7787c160bbb2d35c8"}
Oct 31 23:24:00.075087 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:00.074893    4301 scope.go:122] "RemoveContainer" containerID="fa51886660aad72a3e8fab6906996840346df64773a5627edb1abfd47c6a2609"
Oct 31 23:24:00.076670 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:00.076647    4301 generic.go:356] "Generic (PLEG): container finished" podID="e5c2b8fb-331d-4df2-ad30-13549d8a660c" containerID="222190fd321ae01f361feeaa7c2fe1704ccc18b1be65e3f88b25bbb3ccc70e1f" exitCode=0
Oct 31 23:24:00.076804 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:00.076696    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"
Oct 31 23:24:00.076804 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:00.076722    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz" event={"ID":"e5c2b8fb-331d-4df2-ad30-13549d8a660c","Type":"ContainerDied","Data":"222190fd321ae01f361feeaa7c2fe1704ccc18b1be65e3f88b25bbb3ccc70e1f"}
Oct 31 23:24:00.076804 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:00.076774    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz" event={"ID":"e5c2b8fb-331d-4df2-ad30-13549d8a660c","Type":"ContainerDied","Data":"e0a4965898830da0e2e558d0e37193b4fa76338aa1a9b654567e72147f670545"}
Oct 31 23:24:00.082402 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:00.078825    4301 generic.go:356] "Generic (PLEG): container finished" podID="46881476-3103-4ee9-9b90-ec0ba9f30c3e" containerID="f27c427c52e9908a305179ea8354e0690e33ac07fa37c31b64fe479465bd5033" exitCode=0
Oct 31 23:24:00.082402 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:00.078879    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"
Oct 31 23:24:00.082402 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:00.078904    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7" event={"ID":"46881476-3103-4ee9-9b90-ec0ba9f30c3e","Type":"ContainerDied","Data":"f27c427c52e9908a305179ea8354e0690e33ac07fa37c31b64fe479465bd5033"}
Oct 31 23:24:00.082402 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:00.078962    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7" event={"ID":"46881476-3103-4ee9-9b90-ec0ba9f30c3e","Type":"ContainerDied","Data":"2186d97fd07165f3bd0d10061bc6afb43157b53884132faf92107d64623f2ed9"}
Oct 31 23:24:00.093141 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:00.093119    4301 scope.go:122] "RemoveContainer" containerID="fa51886660aad72a3e8fab6906996840346df64773a5627edb1abfd47c6a2609"
Oct 31 23:24:00.093863 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:24:00.093587    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa51886660aad72a3e8fab6906996840346df64773a5627edb1abfd47c6a2609\": container with ID starting with fa51886660aad72a3e8fab6906996840346df64773a5627edb1abfd47c6a2609 not found: ID does not exist" containerID="fa51886660aad72a3e8fab6906996840346df64773a5627edb1abfd47c6a2609"
Oct 31 23:24:00.093863 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:00.093625    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa51886660aad72a3e8fab6906996840346df64773a5627edb1abfd47c6a2609"} err="failed to get container status \"fa51886660aad72a3e8fab6906996840346df64773a5627edb1abfd47c6a2609\": rpc error: code = NotFound desc = could not find container \"fa51886660aad72a3e8fab6906996840346df64773a5627edb1abfd47c6a2609\": container with ID starting with fa51886660aad72a3e8fab6906996840346df64773a5627edb1abfd47c6a2609 not found: ID does not exist"
Oct 31 23:24:00.093863 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:00.093661    4301 scope.go:122] "RemoveContainer" containerID="222190fd321ae01f361feeaa7c2fe1704ccc18b1be65e3f88b25bbb3ccc70e1f"
Oct 31 23:24:00.104716 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:00.104698    4301 scope.go:122] "RemoveContainer" containerID="222190fd321ae01f361feeaa7c2fe1704ccc18b1be65e3f88b25bbb3ccc70e1f"
Oct 31 23:24:00.105029 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:24:00.105009    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"222190fd321ae01f361feeaa7c2fe1704ccc18b1be65e3f88b25bbb3ccc70e1f\": container with ID starting with 222190fd321ae01f361feeaa7c2fe1704ccc18b1be65e3f88b25bbb3ccc70e1f not found: ID does not exist" containerID="222190fd321ae01f361feeaa7c2fe1704ccc18b1be65e3f88b25bbb3ccc70e1f"
Oct 31 23:24:00.105091 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:00.105037    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"222190fd321ae01f361feeaa7c2fe1704ccc18b1be65e3f88b25bbb3ccc70e1f"} err="failed to get container status \"222190fd321ae01f361feeaa7c2fe1704ccc18b1be65e3f88b25bbb3ccc70e1f\": rpc error: code = NotFound desc = could not find container \"222190fd321ae01f361feeaa7c2fe1704ccc18b1be65e3f88b25bbb3ccc70e1f\": container with ID starting with 222190fd321ae01f361feeaa7c2fe1704ccc18b1be65e3f88b25bbb3ccc70e1f not found: ID does not exist"
Oct 31 23:24:00.105091 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:00.105057    4301 scope.go:122] "RemoveContainer" containerID="f27c427c52e9908a305179ea8354e0690e33ac07fa37c31b64fe479465bd5033"
Oct 31 23:24:00.111469 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:00.111419    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"]
Oct 31 23:24:00.117056 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:00.117031    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-dg6fh"]
Oct 31 23:24:00.119501 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:00.117399    4301 scope.go:122] "RemoveContainer" containerID="876cb199f3051d7a7d6abea4aba1b81d36582ea4122d5e6e269e4ca44106574d"
Oct 31 23:24:00.129328 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:00.129303    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"]
Oct 31 23:24:00.131568 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:00.131552    4301 scope.go:122] "RemoveContainer" containerID="f27c427c52e9908a305179ea8354e0690e33ac07fa37c31b64fe479465bd5033"
Oct 31 23:24:00.131901 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:24:00.131883    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f27c427c52e9908a305179ea8354e0690e33ac07fa37c31b64fe479465bd5033\": container with ID starting with f27c427c52e9908a305179ea8354e0690e33ac07fa37c31b64fe479465bd5033 not found: ID does not exist" containerID="f27c427c52e9908a305179ea8354e0690e33ac07fa37c31b64fe479465bd5033"
Oct 31 23:24:00.131965 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:00.131907    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f27c427c52e9908a305179ea8354e0690e33ac07fa37c31b64fe479465bd5033"} err="failed to get container status \"f27c427c52e9908a305179ea8354e0690e33ac07fa37c31b64fe479465bd5033\": rpc error: code = NotFound desc = could not find container \"f27c427c52e9908a305179ea8354e0690e33ac07fa37c31b64fe479465bd5033\": container with ID starting with f27c427c52e9908a305179ea8354e0690e33ac07fa37c31b64fe479465bd5033 not found: ID does not exist"
Oct 31 23:24:00.131965 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:00.131926    4301 scope.go:122] "RemoveContainer" containerID="876cb199f3051d7a7d6abea4aba1b81d36582ea4122d5e6e269e4ca44106574d"
Oct 31 23:24:00.132240 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:24:00.132224    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"876cb199f3051d7a7d6abea4aba1b81d36582ea4122d5e6e269e4ca44106574d\": container with ID starting with 876cb199f3051d7a7d6abea4aba1b81d36582ea4122d5e6e269e4ca44106574d not found: ID does not exist" containerID="876cb199f3051d7a7d6abea4aba1b81d36582ea4122d5e6e269e4ca44106574d"
Oct 31 23:24:00.132310 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:00.132244    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"876cb199f3051d7a7d6abea4aba1b81d36582ea4122d5e6e269e4ca44106574d"} err="failed to get container status \"876cb199f3051d7a7d6abea4aba1b81d36582ea4122d5e6e269e4ca44106574d\": rpc error: code = NotFound desc = could not find container \"876cb199f3051d7a7d6abea4aba1b81d36582ea4122d5e6e269e4ca44106574d\": container with ID starting with 876cb199f3051d7a7d6abea4aba1b81d36582ea4122d5e6e269e4ca44106574d not found: ID does not exist"
Oct 31 23:24:00.133555 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:00.133537    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-8wtgz"]
Oct 31 23:24:00.141710 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:00.141687    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"]
Oct 31 23:24:00.147964 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:00.147941    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-kgqx7"]
Oct 31 23:24:01.406647 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:01.406617    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46881476-3103-4ee9-9b90-ec0ba9f30c3e" path="/var/lib/kubelet/pods/46881476-3103-4ee9-9b90-ec0ba9f30c3e/volumes"
Oct 31 23:24:01.407138 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:01.407124    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1a1157d-88ea-448f-931d-9528c5df5684" path="/var/lib/kubelet/pods/b1a1157d-88ea-448f-931d-9528c5df5684/volumes"
Oct 31 23:24:01.407572 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:01.407559    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5c2b8fb-331d-4df2-ad30-13549d8a660c" path="/var/lib/kubelet/pods/e5c2b8fb-331d-4df2-ad30-13549d8a660c/volumes"
Oct 31 23:24:02.157573 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:02.157531    4301 patch_prober.go:29] interesting pod/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 23:24:02.157810 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:02.157603    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds" podUID="e9de1723-3c15-4c53-8080-26e27ac118e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:24:02.559787 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:24:02.559702    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:24:04.197794 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:04.197759    4301 generic.go:356] "Generic (PLEG): container finished" podID="e9de1723-3c15-4c53-8080-26e27ac118e4" containerID="b5a8fc8b2e0385dddba02178f1b292286a99189e348c99bf202d5738c63fbb6a" exitCode=0
Oct 31 23:24:04.198260 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:04.197832    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds" event={"ID":"e9de1723-3c15-4c53-8080-26e27ac118e4","Type":"ContainerDied","Data":"b5a8fc8b2e0385dddba02178f1b292286a99189e348c99bf202d5738c63fbb6a"}
Oct 31 23:24:04.198316 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:04.198296    4301 scope.go:122] "RemoveContainer" containerID="b5a8fc8b2e0385dddba02178f1b292286a99189e348c99bf202d5738c63fbb6a"
Oct 31 23:24:04.200123 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:04.200093    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:24:04.200259 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:04.200181    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:24:04.200259 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:04.200230    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:24:05.017295 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:05.017256    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1deb6-mmcrn"]
Oct 31 23:24:05.021091 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:05.021060    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1deb6-mmcrn"]
Oct 31 23:24:05.707421 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:05.657674    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aa97ea3-555d-4b46-9b55-7e55c7481e0b" path="/var/lib/kubelet/pods/7aa97ea3-555d-4b46-9b55-7e55c7481e0b/volumes"
Oct 31 23:24:05.707421 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:05.706849    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds" event={"ID":"e9de1723-3c15-4c53-8080-26e27ac118e4","Type":"ContainerStarted","Data":"192f6f07478d7ec10ff749b7697be9c1ba196660d01b7e03fb5f05cefc0cff0b"}
Oct 31 23:24:06.302381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:06.299137    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:24:06.332387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:06.328330    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:24:06.404439 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:06.368701    4301 patch_prober.go:29] interesting pod/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 23:24:06.404439 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:06.368785    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds" podUID="e9de1723-3c15-4c53-8080-26e27ac118e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 23:24:06.506580 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:06.457511    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:24:06.620133 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:06.543936    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:24:06.620133 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:06.544789    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 31 23:24:06.620133 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:06.553577    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:24:06.620133 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:06.600557    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:24:06.652389 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:06.652323    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:24:06.702839 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:06.702568    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"]
Oct 31 23:24:06.936044 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:06.935820    4301 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:24:06.943771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:06.939393    4301 patch_prober.go:29] interesting pod/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 23:24:06.943771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:06.939444    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds" podUID="e9de1723-3c15-4c53-8080-26e27ac118e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 23:24:06.948789 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:06.948754    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6" podUID="8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2" containerName="istio-proxy" containerID="cri-o://97b51d48183b8e6ee01153c6871cd74b0c86ab1ad7251bb37a7669a07445a75f" gracePeriod=30
Oct 31 23:24:06.958842 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:06.958804    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"]
Oct 31 23:24:06.959301 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:06.959267    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f" podUID="f8595a54-27b2-43a5-9e7d-f7d0fd41c523" containerName="istio-proxy" containerID="cri-o://cc3499284872061224ffd6f190018441fa71576d01dc5c1c344cc6425e0ca619" gracePeriod=30
Oct 31 23:24:07.828030 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:07.827994    4301 patch_prober.go:29] interesting pod/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 23:24:07.828235 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:07.828055    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds" podUID="e9de1723-3c15-4c53-8080-26e27ac118e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 23:24:08.827086 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:08.827052    4301 patch_prober.go:29] interesting pod/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 23:24:08.827533 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:08.827100    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds" podUID="e9de1723-3c15-4c53-8080-26e27ac118e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 23:24:09.826969 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:09.826933    4301 patch_prober.go:29] interesting pod/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 23:24:09.827182 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:09.826999    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds" podUID="e9de1723-3c15-4c53-8080-26e27ac118e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 23:24:10.480071 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:10.479421    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2126ff-7gqc9"]
Oct 31 23:24:10.591574 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:10.556201    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2126ff-7gqc9"]
Oct 31 23:24:10.827471 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:10.827390    4301 patch_prober.go:29] interesting pod/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 23:24:10.827471 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:10.827449    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds" podUID="e9de1723-3c15-4c53-8080-26e27ac118e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 23:24:11.501314 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:11.501281    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a0fbeac-3a35-47cc-89fe-dcd510841e18" path="/var/lib/kubelet/pods/0a0fbeac-3a35-47cc-89fe-dcd510841e18/volumes"
Oct 31 23:24:11.925094 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:11.925056    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:24:11.925681 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:11.925665    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:24:11.926760 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:11.926741    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:24:12.174734 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.174697    4301 generic.go:356] "Generic (PLEG): container finished" podID="8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2" containerID="97b51d48183b8e6ee01153c6871cd74b0c86ab1ad7251bb37a7669a07445a75f" exitCode=0
Oct 31 23:24:12.176650 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.176495    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6" event={"ID":"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2","Type":"ContainerDied","Data":"97b51d48183b8e6ee01153c6871cd74b0c86ab1ad7251bb37a7669a07445a75f"}
Oct 31 23:24:12.360937 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.360912    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:24:12.529530 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.529485    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-credential-socket\") pod \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") "
Oct 31 23:24:12.529737 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.529659    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istio-token\" (UniqueName: \"kubernetes.io/projected/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istio-token\") pod \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") "
Oct 31 23:24:12.529737 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.529695    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-workload-socket\") pod \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") "
Oct 31 23:24:12.529841 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.529754    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-kube-api-access-p9zm8\" (UniqueName: \"kubernetes.io/projected/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-kube-api-access-p9zm8\") pod \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") "
Oct 31 23:24:12.529841 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.529790    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-workload-certs\") pod \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") "
Oct 31 23:24:12.529920 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.529861    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istio-podinfo\") pod \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") "
Oct 31 23:24:12.529920 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.529907    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istio-envoy\") pod \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") "
Oct 31 23:24:12.530663 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.530023    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istiod-ca-cert\") pod \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") "
Oct 31 23:24:12.530663 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.530111    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istio-data\") pod \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\" (UID: \"f8595a54-27b2-43a5-9e7d-f7d0fd41c523\") "
Oct 31 23:24:12.530663 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.530290    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-workload-socket" pod "f8595a54-27b2-43a5-9e7d-f7d0fd41c523" (UID: "f8595a54-27b2-43a5-9e7d-f7d0fd41c523"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:12.530663 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.530337    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-workload-certs" pod "f8595a54-27b2-43a5-9e7d-f7d0fd41c523" (UID: "f8595a54-27b2-43a5-9e7d-f7d0fd41c523"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:12.530663 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.530427    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-credential-socket" pod "f8595a54-27b2-43a5-9e7d-f7d0fd41c523" (UID: "f8595a54-27b2-43a5-9e7d-f7d0fd41c523"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:12.531012 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.530723    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istio-data" pod "f8595a54-27b2-43a5-9e7d-f7d0fd41c523" (UID: "f8595a54-27b2-43a5-9e7d-f7d0fd41c523"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:12.531012 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.530692    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istiod-ca-cert" pod "f8595a54-27b2-43a5-9e7d-f7d0fd41c523" (UID: "f8595a54-27b2-43a5-9e7d-f7d0fd41c523"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:24:12.531012 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.530879    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:12.531012 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.530898    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:12.531012 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.530912    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:12.531012 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.530926    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:12.531012 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.530940    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:12.532310 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.532283    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istio-token" pod "f8595a54-27b2-43a5-9e7d-f7d0fd41c523" (UID: "f8595a54-27b2-43a5-9e7d-f7d0fd41c523"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:12.532645 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.532617    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-kube-api-access-p9zm8" pod "f8595a54-27b2-43a5-9e7d-f7d0fd41c523" (UID: "f8595a54-27b2-43a5-9e7d-f7d0fd41c523"). InnerVolumeSpecName "kube-api-access-p9zm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:12.532765 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.532738    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istio-podinfo" pod "f8595a54-27b2-43a5-9e7d-f7d0fd41c523" (UID: "f8595a54-27b2-43a5-9e7d-f7d0fd41c523"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:24:12.533525 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.533505    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istio-envoy" pod "f8595a54-27b2-43a5-9e7d-f7d0fd41c523" (UID: "f8595a54-27b2-43a5-9e7d-f7d0fd41c523"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:12.632636 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.632597    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:12.632636 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.632633    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9zm8\" (UniqueName: \"kubernetes.io/projected/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-kube-api-access-p9zm8\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:12.632636 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.632644    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:12.632936 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.632653    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8595a54-27b2-43a5-9e7d-f7d0fd41c523-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:12.662433 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.662404    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:24:12.799301 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.799203    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-workload-socket\") pod \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") "
Oct 31 23:24:12.799301 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.799254    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-credential-socket\") pod \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") "
Oct 31 23:24:12.799607 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.799320    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istiod-ca-cert\") pod \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") "
Oct 31 23:24:12.799607 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.799410    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istio-podinfo\") pod \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") "
Oct 31 23:24:12.799607 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.799475    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istio-envoy\") pod \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") "
Oct 31 23:24:12.799607 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.799500    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istio-token\" (UniqueName: \"kubernetes.io/projected/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istio-token\") pod \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") "
Oct 31 23:24:12.799825 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.799614    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-workload-certs\") pod \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") "
Oct 31 23:24:12.799825 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.799664    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istio-data\") pod \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") "
Oct 31 23:24:12.799825 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.799759    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-kube-api-access-qzr92\" (UniqueName: \"kubernetes.io/projected/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-kube-api-access-qzr92\") pod \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\" (UID: \"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2\") "
Oct 31 23:24:12.831022 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.830984    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-workload-socket" pod "8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2" (UID: "8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:12.831499 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.831472    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-credential-socket" pod "8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2" (UID: "8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:12.838066 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.837998    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istiod-ca-cert" pod "8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2" (UID: "8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:24:12.838711 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.838645    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-workload-certs" pod "8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2" (UID: "8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:12.855425 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.850451    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istio-data" pod "8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2" (UID: "8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:12.927380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.916935    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istio-token" pod "8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2" (UID: "8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:12.939666 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.938191    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:12.939666 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.938232    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:12.939666 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.938250    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:12.939666 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.938264    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:12.939666 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.938278    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:12.939666 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.938292    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:12.952702 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.952641    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-kube-api-access-qzr92" pod "8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2" (UID: "8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2"). InnerVolumeSpecName "kube-api-access-qzr92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:12.953030 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.953005    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istio-envoy" pod "8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2" (UID: "8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:12.953245 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:12.953199    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istio-podinfo" pod "8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2" (UID: "8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:24:13.125685 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:13.125655    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzr92\" (UniqueName: \"kubernetes.io/projected/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-kube-api-access-qzr92\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:13.125858 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:13.125847    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:13.125944 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:13.125934    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:13.183811 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:13.183781    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"
Oct 31 23:24:13.183989 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:13.183787    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6" event={"ID":"8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2","Type":"ContainerDied","Data":"6133ed3031ee5d16e93ceb59667becd63f2c36dc8afcc8d138cf93d3f43bd4d7"}
Oct 31 23:24:13.183989 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:13.183913    4301 scope.go:122] "RemoveContainer" containerID="97b51d48183b8e6ee01153c6871cd74b0c86ab1ad7251bb37a7669a07445a75f"
Oct 31 23:24:13.185498 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:13.185471    4301 generic.go:356] "Generic (PLEG): container finished" podID="f8595a54-27b2-43a5-9e7d-f7d0fd41c523" containerID="cc3499284872061224ffd6f190018441fa71576d01dc5c1c344cc6425e0ca619" exitCode=0
Oct 31 23:24:13.185667 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:13.185513    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"
Oct 31 23:24:13.185667 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:13.185552    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f" event={"ID":"f8595a54-27b2-43a5-9e7d-f7d0fd41c523","Type":"ContainerDied","Data":"cc3499284872061224ffd6f190018441fa71576d01dc5c1c344cc6425e0ca619"}
Oct 31 23:24:13.185667 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:13.185637    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f" event={"ID":"f8595a54-27b2-43a5-9e7d-f7d0fd41c523","Type":"ContainerDied","Data":"38365cc7d62a343d5856a2c22523885615a5885bff469f2804757bfb2d96ffb7"}
Oct 31 23:24:13.207879 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:13.203903    4301 scope.go:122] "RemoveContainer" containerID="cc3499284872061224ffd6f190018441fa71576d01dc5c1c344cc6425e0ca619"
Oct 31 23:24:13.224427 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:13.224330    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"]
Oct 31 23:24:13.231158 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:13.231136    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"]
Oct 31 23:24:13.231608 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:13.231577    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck" podUID="1fd21e5d-47da-405b-9724-84cafcd106d2" containerName="istio-proxy" containerID="cri-o://dcbecb35bcaa54742a40464e3698a8b9203ca9a0d259d6cc801813a77e8aec74" gracePeriod=30
Oct 31 23:24:13.236847 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:13.236823    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"]
Oct 31 23:24:13.237321 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:13.237255    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd" podUID="34836e36-7a9c-4072-a555-970e2eb17c2b" containerName="istio-proxy" containerID="cri-o://be25f67eed079add94747dc1a101d51b53b0d7fd3400cd273204bc930b358323" gracePeriod=30
Oct 31 23:24:13.245032 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:13.244969    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-8k2b6"]
Oct 31 23:24:13.248845 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:13.248373    4301 scope.go:122] "RemoveContainer" containerID="cc3499284872061224ffd6f190018441fa71576d01dc5c1c344cc6425e0ca619"
Oct 31 23:24:13.248845 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:24:13.248759    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc3499284872061224ffd6f190018441fa71576d01dc5c1c344cc6425e0ca619\": container with ID starting with cc3499284872061224ffd6f190018441fa71576d01dc5c1c344cc6425e0ca619 not found: ID does not exist" containerID="cc3499284872061224ffd6f190018441fa71576d01dc5c1c344cc6425e0ca619"
Oct 31 23:24:13.248845 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:13.248790    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc3499284872061224ffd6f190018441fa71576d01dc5c1c344cc6425e0ca619"} err="failed to get container status \"cc3499284872061224ffd6f190018441fa71576d01dc5c1c344cc6425e0ca619\": rpc error: code = NotFound desc = could not find container \"cc3499284872061224ffd6f190018441fa71576d01dc5c1c344cc6425e0ca619\": container with ID starting with cc3499284872061224ffd6f190018441fa71576d01dc5c1c344cc6425e0ca619 not found: ID does not exist"
Oct 31 23:24:13.268545 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:13.268496    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"]
Oct 31 23:24:13.274329 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:13.274305    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-brh5f"]
Oct 31 23:24:13.563425 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:13.558437    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2" path="/var/lib/kubelet/pods/8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2/volumes"
Oct 31 23:24:13.582489 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:13.582223    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8595a54-27b2-43a5-9e7d-f7d0fd41c523" path="/var/lib/kubelet/pods/f8595a54-27b2-43a5-9e7d-f7d0fd41c523/volumes"
Oct 31 23:24:15.233504 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:15.233465    4301 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 23:24:15.233912 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:15.233521    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd" podUID="34836e36-7a9c-4072-a555-970e2eb17c2b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 23:24:15.446243 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:15.446211    4301 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 23:24:15.446415 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:15.446261    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck" podUID="1fd21e5d-47da-405b-9724-84cafcd106d2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 23:24:16.330250 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:16.330215    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d6685e-7c669d9694-8jd86"]
Oct 31 23:24:16.344108 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:16.344074    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6685e-7bdfff8567-s9nqv"]
Oct 31 23:24:16.715387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:16.701418    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f80dfa-847989bcd7-dcg6p"]
Oct 31 23:24:18.748768 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.748732    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:24:18.752191 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.752167    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:24:18.885874 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.885838    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-credential-socket\") pod \"34836e36-7a9c-4072-a555-970e2eb17c2b\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") "
Oct 31 23:24:18.886071 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.885878    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-istio-data\") pod \"34836e36-7a9c-4072-a555-970e2eb17c2b\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") "
Oct 31 23:24:18.886071 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.885931    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34836e36-7a9c-4072-a555-970e2eb17c2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34836e36-7a9c-4072-a555-970e2eb17c2b-istiod-ca-cert\") pod \"34836e36-7a9c-4072-a555-970e2eb17c2b\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") "
Oct 31 23:24:18.886071 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.885965    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34836e36-7a9c-4072-a555-970e2eb17c2b-kube-api-access-x2t44\" (UniqueName: \"kubernetes.io/projected/34836e36-7a9c-4072-a555-970e2eb17c2b-kube-api-access-x2t44\") pod \"34836e36-7a9c-4072-a555-970e2eb17c2b\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") "
Oct 31 23:24:18.886071 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.885998    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34836e36-7a9c-4072-a555-970e2eb17c2b-istio-token\" (UniqueName: \"kubernetes.io/projected/34836e36-7a9c-4072-a555-970e2eb17c2b-istio-token\") pod \"34836e36-7a9c-4072-a555-970e2eb17c2b\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") "
Oct 31 23:24:18.886071 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.886025    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-workload-socket\") pod \"34836e36-7a9c-4072-a555-970e2eb17c2b\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") "
Oct 31 23:24:18.886413 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.886094    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fd21e5d-47da-405b-9724-84cafcd106d2-istio-token\" (UniqueName: \"kubernetes.io/projected/1fd21e5d-47da-405b-9724-84cafcd106d2-istio-token\") pod \"1fd21e5d-47da-405b-9724-84cafcd106d2\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") "
Oct 31 23:24:18.886413 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.886097    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-credential-socket" pod "34836e36-7a9c-4072-a555-970e2eb17c2b" (UID: "34836e36-7a9c-4072-a555-970e2eb17c2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:18.886413 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.886198    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fd21e5d-47da-405b-9724-84cafcd106d2-kube-api-access-zh9dl\" (UniqueName: \"kubernetes.io/projected/1fd21e5d-47da-405b-9724-84cafcd106d2-kube-api-access-zh9dl\") pod \"1fd21e5d-47da-405b-9724-84cafcd106d2\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") "
Oct 31 23:24:18.886413 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.886205    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-istio-data" pod "34836e36-7a9c-4072-a555-970e2eb17c2b" (UID: "34836e36-7a9c-4072-a555-970e2eb17c2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:18.886413 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.886280    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-workload-socket\") pod \"1fd21e5d-47da-405b-9724-84cafcd106d2\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") "
Oct 31 23:24:18.886722 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.886446    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-istio-data\") pod \"1fd21e5d-47da-405b-9724-84cafcd106d2\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") "
Oct 31 23:24:18.886722 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.886452    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-workload-socket" pod "34836e36-7a9c-4072-a555-970e2eb17c2b" (UID: "34836e36-7a9c-4072-a555-970e2eb17c2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:18.886722 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.886502    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-credential-socket\") pod \"1fd21e5d-47da-405b-9724-84cafcd106d2\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") "
Oct 31 23:24:18.886722 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.886544    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-workload-certs\") pod \"34836e36-7a9c-4072-a555-970e2eb17c2b\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") "
Oct 31 23:24:18.886722 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.886590    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fd21e5d-47da-405b-9724-84cafcd106d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fd21e5d-47da-405b-9724-84cafcd106d2-istio-podinfo\") pod \"1fd21e5d-47da-405b-9724-84cafcd106d2\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") "
Oct 31 23:24:18.886722 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.886504    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34836e36-7a9c-4072-a555-970e2eb17c2b-istiod-ca-cert" pod "34836e36-7a9c-4072-a555-970e2eb17c2b" (UID: "34836e36-7a9c-4072-a555-970e2eb17c2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:24:18.886722 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.886661    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34836e36-7a9c-4072-a555-970e2eb17c2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34836e36-7a9c-4072-a555-970e2eb17c2b-istio-podinfo\") pod \"34836e36-7a9c-4072-a555-970e2eb17c2b\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") "
Oct 31 23:24:18.886722 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.886663    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-workload-socket" pod "1fd21e5d-47da-405b-9724-84cafcd106d2" (UID: "1fd21e5d-47da-405b-9724-84cafcd106d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:18.886722 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.886691    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-istio-data" pod "1fd21e5d-47da-405b-9724-84cafcd106d2" (UID: "1fd21e5d-47da-405b-9724-84cafcd106d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:18.886722 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.886695    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-istio-envoy\") pod \"34836e36-7a9c-4072-a555-970e2eb17c2b\" (UID: \"34836e36-7a9c-4072-a555-970e2eb17c2b\") "
Oct 31 23:24:18.887229 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.886755    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fd21e5d-47da-405b-9724-84cafcd106d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fd21e5d-47da-405b-9724-84cafcd106d2-istiod-ca-cert\") pod \"1fd21e5d-47da-405b-9724-84cafcd106d2\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") "
Oct 31 23:24:18.887229 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.886827    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-credential-socket" pod "1fd21e5d-47da-405b-9724-84cafcd106d2" (UID: "1fd21e5d-47da-405b-9724-84cafcd106d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:18.887229 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.886833    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-workload-certs\") pod \"1fd21e5d-47da-405b-9724-84cafcd106d2\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") "
Oct 31 23:24:18.887229 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.886872    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-workload-certs" pod "34836e36-7a9c-4072-a555-970e2eb17c2b" (UID: "34836e36-7a9c-4072-a555-970e2eb17c2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:18.887229 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.886929    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-istio-envoy\") pod \"1fd21e5d-47da-405b-9724-84cafcd106d2\" (UID: \"1fd21e5d-47da-405b-9724-84cafcd106d2\") "
Oct 31 23:24:18.887229 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.886985    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-workload-certs" pod "1fd21e5d-47da-405b-9724-84cafcd106d2" (UID: "1fd21e5d-47da-405b-9724-84cafcd106d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:18.887848 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.887821    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:18.887936 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.887852    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:18.887936 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.887867    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34836e36-7a9c-4072-a555-970e2eb17c2b-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:18.887936 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.887919    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:18.888109 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.887937    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:18.888109 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.887952    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:18.888109 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.887967    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:18.888109 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.887983    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:18.888109 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.887998    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:18.889118 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.888474    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fd21e5d-47da-405b-9724-84cafcd106d2-istiod-ca-cert" pod "1fd21e5d-47da-405b-9724-84cafcd106d2" (UID: "1fd21e5d-47da-405b-9724-84cafcd106d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:24:18.889118 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.888839    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fd21e5d-47da-405b-9724-84cafcd106d2-istio-token" pod "1fd21e5d-47da-405b-9724-84cafcd106d2" (UID: "1fd21e5d-47da-405b-9724-84cafcd106d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:18.889118 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.888879    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34836e36-7a9c-4072-a555-970e2eb17c2b-kube-api-access-x2t44" pod "34836e36-7a9c-4072-a555-970e2eb17c2b" (UID: "34836e36-7a9c-4072-a555-970e2eb17c2b"). InnerVolumeSpecName "kube-api-access-x2t44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:18.889118 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.889023    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34836e36-7a9c-4072-a555-970e2eb17c2b-istio-token" pod "34836e36-7a9c-4072-a555-970e2eb17c2b" (UID: "34836e36-7a9c-4072-a555-970e2eb17c2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:18.889349 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.889170    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-istio-envoy" pod "34836e36-7a9c-4072-a555-970e2eb17c2b" (UID: "34836e36-7a9c-4072-a555-970e2eb17c2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:18.889451 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.889431    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-istio-envoy" pod "1fd21e5d-47da-405b-9724-84cafcd106d2" (UID: "1fd21e5d-47da-405b-9724-84cafcd106d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:18.889659 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.889644    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fd21e5d-47da-405b-9724-84cafcd106d2-istio-podinfo" pod "1fd21e5d-47da-405b-9724-84cafcd106d2" (UID: "1fd21e5d-47da-405b-9724-84cafcd106d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:24:18.890195 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.890177    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fd21e5d-47da-405b-9724-84cafcd106d2-kube-api-access-zh9dl" pod "1fd21e5d-47da-405b-9724-84cafcd106d2" (UID: "1fd21e5d-47da-405b-9724-84cafcd106d2"). InnerVolumeSpecName "kube-api-access-zh9dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:18.890473 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.890450    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34836e36-7a9c-4072-a555-970e2eb17c2b-istio-podinfo" pod "34836e36-7a9c-4072-a555-970e2eb17c2b" (UID: "34836e36-7a9c-4072-a555-970e2eb17c2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:24:18.989197 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.989159    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fd21e5d-47da-405b-9724-84cafcd106d2-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:18.989197 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.989189    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2t44\" (UniqueName: \"kubernetes.io/projected/34836e36-7a9c-4072-a555-970e2eb17c2b-kube-api-access-x2t44\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:18.989197 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.989201    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34836e36-7a9c-4072-a555-970e2eb17c2b-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:18.989553 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.989210    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fd21e5d-47da-405b-9724-84cafcd106d2-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:18.989553 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.989220    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh9dl\" (UniqueName: \"kubernetes.io/projected/1fd21e5d-47da-405b-9724-84cafcd106d2-kube-api-access-zh9dl\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:18.989553 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.989228    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fd21e5d-47da-405b-9724-84cafcd106d2-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:18.989553 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.989237    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34836e36-7a9c-4072-a555-970e2eb17c2b-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:18.989553 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.989246    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34836e36-7a9c-4072-a555-970e2eb17c2b-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:18.989553 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:18.989254    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fd21e5d-47da-405b-9724-84cafcd106d2-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:19.317134 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:19.317036    4301 generic.go:356] "Generic (PLEG): container finished" podID="34836e36-7a9c-4072-a555-970e2eb17c2b" containerID="be25f67eed079add94747dc1a101d51b53b0d7fd3400cd273204bc930b358323" exitCode=0
Oct 31 23:24:19.317310 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:19.317192    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd" event={"ID":"34836e36-7a9c-4072-a555-970e2eb17c2b","Type":"ContainerDied","Data":"be25f67eed079add94747dc1a101d51b53b0d7fd3400cd273204bc930b358323"}
Oct 31 23:24:19.317310 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:19.317247    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd" event={"ID":"34836e36-7a9c-4072-a555-970e2eb17c2b","Type":"ContainerDied","Data":"ddd0608a9b54036977598ddb2b2e7df67e39e8065a1608c97e22d69bab076a6d"}
Oct 31 23:24:19.317310 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:19.317274    4301 scope.go:122] "RemoveContainer" containerID="be25f67eed079add94747dc1a101d51b53b0d7fd3400cd273204bc930b358323"
Oct 31 23:24:19.317663 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:19.317463    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"
Oct 31 23:24:19.337424 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:19.328309    4301 generic.go:356] "Generic (PLEG): container finished" podID="1fd21e5d-47da-405b-9724-84cafcd106d2" containerID="dcbecb35bcaa54742a40464e3698a8b9203ca9a0d259d6cc801813a77e8aec74" exitCode=0
Oct 31 23:24:19.337424 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:19.328605    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"
Oct 31 23:24:19.337424 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:19.331428    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck" event={"ID":"1fd21e5d-47da-405b-9724-84cafcd106d2","Type":"ContainerDied","Data":"dcbecb35bcaa54742a40464e3698a8b9203ca9a0d259d6cc801813a77e8aec74"}
Oct 31 23:24:19.337424 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:19.331474    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck" event={"ID":"1fd21e5d-47da-405b-9724-84cafcd106d2","Type":"ContainerDied","Data":"80a7e196fb1e99222e6ed04809e96aee6f8573d5d94b8802fbdae6390319faca"}
Oct 31 23:24:19.341132 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:19.341112    4301 scope.go:122] "RemoveContainer" containerID="be25f67eed079add94747dc1a101d51b53b0d7fd3400cd273204bc930b358323"
Oct 31 23:24:19.341477 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:24:19.341443    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be25f67eed079add94747dc1a101d51b53b0d7fd3400cd273204bc930b358323\": container with ID starting with be25f67eed079add94747dc1a101d51b53b0d7fd3400cd273204bc930b358323 not found: ID does not exist" containerID="be25f67eed079add94747dc1a101d51b53b0d7fd3400cd273204bc930b358323"
Oct 31 23:24:19.341578 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:19.341478    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be25f67eed079add94747dc1a101d51b53b0d7fd3400cd273204bc930b358323"} err="failed to get container status \"be25f67eed079add94747dc1a101d51b53b0d7fd3400cd273204bc930b358323\": rpc error: code = NotFound desc = could not find container \"be25f67eed079add94747dc1a101d51b53b0d7fd3400cd273204bc930b358323\": container with ID starting with be25f67eed079add94747dc1a101d51b53b0d7fd3400cd273204bc930b358323 not found: ID does not exist"
Oct 31 23:24:19.341578 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:19.341495    4301 scope.go:122] "RemoveContainer" containerID="dcbecb35bcaa54742a40464e3698a8b9203ca9a0d259d6cc801813a77e8aec74"
Oct 31 23:24:19.352561 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:19.352522    4301 scope.go:122] "RemoveContainer" containerID="dcbecb35bcaa54742a40464e3698a8b9203ca9a0d259d6cc801813a77e8aec74"
Oct 31 23:24:19.352844 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:24:19.352824    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcbecb35bcaa54742a40464e3698a8b9203ca9a0d259d6cc801813a77e8aec74\": container with ID starting with dcbecb35bcaa54742a40464e3698a8b9203ca9a0d259d6cc801813a77e8aec74 not found: ID does not exist" containerID="dcbecb35bcaa54742a40464e3698a8b9203ca9a0d259d6cc801813a77e8aec74"
Oct 31 23:24:19.352918 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:19.352851    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcbecb35bcaa54742a40464e3698a8b9203ca9a0d259d6cc801813a77e8aec74"} err="failed to get container status \"dcbecb35bcaa54742a40464e3698a8b9203ca9a0d259d6cc801813a77e8aec74\": rpc error: code = NotFound desc = could not find container \"dcbecb35bcaa54742a40464e3698a8b9203ca9a0d259d6cc801813a77e8aec74\": container with ID starting with dcbecb35bcaa54742a40464e3698a8b9203ca9a0d259d6cc801813a77e8aec74 not found: ID does not exist"
Oct 31 23:24:19.363400 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:19.363376    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"]
Oct 31 23:24:19.368964 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:19.368939    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-kp7xd"]
Oct 31 23:24:19.376975 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:19.376953    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"]
Oct 31 23:24:19.385440 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:19.385420    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-z8wck"]
Oct 31 23:24:19.405554 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:19.405527    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fd21e5d-47da-405b-9724-84cafcd106d2" path="/var/lib/kubelet/pods/1fd21e5d-47da-405b-9724-84cafcd106d2/volumes"
Oct 31 23:24:19.406120 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:19.406098    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34836e36-7a9c-4072-a555-970e2eb17c2b" path="/var/lib/kubelet/pods/34836e36-7a9c-4072-a555-970e2eb17c2b/volumes"
Oct 31 23:24:21.335427 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:21.331431    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6685e-7c669d9694-8jd86" podUID="b9eac259-b8e2-4fff-a019-2daf3c17c47b" containerName="payload-processing" containerID="cri-o://57d9d967eb5a8fe9fb27f47d3ebf2d1dbd43d04c7d4858ec17807d9dd41e58e0" gracePeriod=25
Oct 31 23:24:21.350587 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:21.344454    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6685e-7bdfff8567-s9nqv" podUID="35ceb2a4-953d-4719-9d85-e3c8491d1825" containerName="payload-pre-processing" containerID="cri-o://49034d96749eaaea772cacd57f65025100f34fae047aeb6bfec1a05d795eb028" gracePeriod=25
Oct 31 23:24:21.496417 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:24:21.495943    4301 logging.go:55] [core] [Channel #1069 SubChannel #1070]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.87:50052", ServerName: "10.133.0.87:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.87:50052: connect: connection refused"
Oct 31 23:24:21.702390 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:21.701768    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f80dfa-847989bcd7-dcg6p" podUID="faee4004-8214-4ca7-9dc4-da1a18620c17" containerName="payload-pre-processing" containerID="cri-o://5e73ef3a2e757e8886446d510f728e56d30c75c9780c3bf40494967422e6084b" gracePeriod=25
Oct 31 23:24:21.955606 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:21.955127    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6685e-7c669d9694-8jd86"
Oct 31 23:24:21.975272 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:21.975180    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6685e-7bdfff8567-s9nqv"
Oct 31 23:24:22.008062 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.008023    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9eac259-b8e2-4fff-a019-2daf3c17c47b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9eac259-b8e2-4fff-a019-2daf3c17c47b-plugins-config-volume\") pod \"b9eac259-b8e2-4fff-a019-2daf3c17c47b\" (UID: \"b9eac259-b8e2-4fff-a019-2daf3c17c47b\") "
Oct 31 23:24:22.008491 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.008464    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9eac259-b8e2-4fff-a019-2daf3c17c47b-kube-api-access-5fn97\" (UniqueName: \"kubernetes.io/projected/b9eac259-b8e2-4fff-a019-2daf3c17c47b-kube-api-access-5fn97\") pod \"b9eac259-b8e2-4fff-a019-2daf3c17c47b\" (UID: \"b9eac259-b8e2-4fff-a019-2daf3c17c47b\") "
Oct 31 23:24:22.011905 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.011874    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9eac259-b8e2-4fff-a019-2daf3c17c47b-kube-api-access-5fn97" pod "b9eac259-b8e2-4fff-a019-2daf3c17c47b" (UID: "b9eac259-b8e2-4fff-a019-2daf3c17c47b"). InnerVolumeSpecName "kube-api-access-5fn97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:22.012237 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.012217    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9eac259-b8e2-4fff-a019-2daf3c17c47b-plugins-config-volume" pod "b9eac259-b8e2-4fff-a019-2daf3c17c47b" (UID: "b9eac259-b8e2-4fff-a019-2daf3c17c47b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:24:22.057221 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.057194    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f80dfa-847989bcd7-dcg6p"
Oct 31 23:24:22.109982 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.109950    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faee4004-8214-4ca7-9dc4-da1a18620c17-kube-api-access-5ls9c\" (UniqueName: \"kubernetes.io/projected/faee4004-8214-4ca7-9dc4-da1a18620c17-kube-api-access-5ls9c\") pod \"faee4004-8214-4ca7-9dc4-da1a18620c17\" (UID: \"faee4004-8214-4ca7-9dc4-da1a18620c17\") "
Oct 31 23:24:22.109982 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.109985    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faee4004-8214-4ca7-9dc4-da1a18620c17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faee4004-8214-4ca7-9dc4-da1a18620c17-plugins-config-volume\") pod \"faee4004-8214-4ca7-9dc4-da1a18620c17\" (UID: \"faee4004-8214-4ca7-9dc4-da1a18620c17\") "
Oct 31 23:24:22.110232 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.110004    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35ceb2a4-953d-4719-9d85-e3c8491d1825-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35ceb2a4-953d-4719-9d85-e3c8491d1825-plugins-config-volume\") pod \"35ceb2a4-953d-4719-9d85-e3c8491d1825\" (UID: \"35ceb2a4-953d-4719-9d85-e3c8491d1825\") "
Oct 31 23:24:22.110232 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.110125    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35ceb2a4-953d-4719-9d85-e3c8491d1825-kube-api-access-qfxng\" (UniqueName: \"kubernetes.io/projected/35ceb2a4-953d-4719-9d85-e3c8491d1825-kube-api-access-qfxng\") pod \"35ceb2a4-953d-4719-9d85-e3c8491d1825\" (UID: \"35ceb2a4-953d-4719-9d85-e3c8491d1825\") "
Oct 31 23:24:22.110522 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.110487    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faee4004-8214-4ca7-9dc4-da1a18620c17-plugins-config-volume" pod "faee4004-8214-4ca7-9dc4-da1a18620c17" (UID: "faee4004-8214-4ca7-9dc4-da1a18620c17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:24:22.110893 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.110865    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35ceb2a4-953d-4719-9d85-e3c8491d1825-plugins-config-volume" pod "35ceb2a4-953d-4719-9d85-e3c8491d1825" (UID: "35ceb2a4-953d-4719-9d85-e3c8491d1825"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:24:22.111014 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.110954    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fn97\" (UniqueName: \"kubernetes.io/projected/b9eac259-b8e2-4fff-a019-2daf3c17c47b-kube-api-access-5fn97\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:22.111014 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.110981    4301 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faee4004-8214-4ca7-9dc4-da1a18620c17-plugins-config-volume\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:22.111014 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.110996    4301 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9eac259-b8e2-4fff-a019-2daf3c17c47b-plugins-config-volume\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:22.112272 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.112251    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35ceb2a4-953d-4719-9d85-e3c8491d1825-kube-api-access-qfxng" pod "35ceb2a4-953d-4719-9d85-e3c8491d1825" (UID: "35ceb2a4-953d-4719-9d85-e3c8491d1825"). InnerVolumeSpecName "kube-api-access-qfxng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:22.112732 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.112715    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faee4004-8214-4ca7-9dc4-da1a18620c17-kube-api-access-5ls9c" pod "faee4004-8214-4ca7-9dc4-da1a18620c17" (UID: "faee4004-8214-4ca7-9dc4-da1a18620c17"). InnerVolumeSpecName "kube-api-access-5ls9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:22.212283 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.212204    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ls9c\" (UniqueName: \"kubernetes.io/projected/faee4004-8214-4ca7-9dc4-da1a18620c17-kube-api-access-5ls9c\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:22.212283 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.212233    4301 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35ceb2a4-953d-4719-9d85-e3c8491d1825-plugins-config-volume\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:22.212283 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.212243    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfxng\" (UniqueName: \"kubernetes.io/projected/35ceb2a4-953d-4719-9d85-e3c8491d1825-kube-api-access-qfxng\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:22.490794 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.490702    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6685e-7bdfff8567-s9nqv" podUID="35ceb2a4-953d-4719-9d85-e3c8491d1825" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.87:50052\" within 1s: context deadline exceeded"
Oct 31 23:24:22.528970 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.528939    4301 generic.go:356] "Generic (PLEG): container finished" podID="35ceb2a4-953d-4719-9d85-e3c8491d1825" containerID="49034d96749eaaea772cacd57f65025100f34fae047aeb6bfec1a05d795eb028" exitCode=0
Oct 31 23:24:22.529142 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.528980    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6685e-7bdfff8567-s9nqv"
Oct 31 23:24:22.529142 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.529017    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6685e-7bdfff8567-s9nqv" event={"ID":"35ceb2a4-953d-4719-9d85-e3c8491d1825","Type":"ContainerDied","Data":"49034d96749eaaea772cacd57f65025100f34fae047aeb6bfec1a05d795eb028"}
Oct 31 23:24:22.529142 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.529075    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6685e-7bdfff8567-s9nqv" event={"ID":"35ceb2a4-953d-4719-9d85-e3c8491d1825","Type":"ContainerDied","Data":"a0127463ea4d8b83946181fbb5b25dbcfce0800e54baa1531929f8765a47c55e"}
Oct 31 23:24:22.529142 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.529105    4301 scope.go:122] "RemoveContainer" containerID="49034d96749eaaea772cacd57f65025100f34fae047aeb6bfec1a05d795eb028"
Oct 31 23:24:22.530842 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.530680    4301 generic.go:356] "Generic (PLEG): container finished" podID="faee4004-8214-4ca7-9dc4-da1a18620c17" containerID="5e73ef3a2e757e8886446d510f728e56d30c75c9780c3bf40494967422e6084b" exitCode=0
Oct 31 23:24:22.530842 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.530709    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f80dfa-847989bcd7-dcg6p" event={"ID":"faee4004-8214-4ca7-9dc4-da1a18620c17","Type":"ContainerDied","Data":"5e73ef3a2e757e8886446d510f728e56d30c75c9780c3bf40494967422e6084b"}
Oct 31 23:24:22.530842 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.530725    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f80dfa-847989bcd7-dcg6p"
Oct 31 23:24:22.530842 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.530764    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f80dfa-847989bcd7-dcg6p" event={"ID":"faee4004-8214-4ca7-9dc4-da1a18620c17","Type":"ContainerDied","Data":"8aeab6b08e8ac279f280f979cc27fccfc1187dcfb1c6503ec88d0197a0e57ad1"}
Oct 31 23:24:22.532555 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.532530    4301 generic.go:356] "Generic (PLEG): container finished" podID="b9eac259-b8e2-4fff-a019-2daf3c17c47b" containerID="57d9d967eb5a8fe9fb27f47d3ebf2d1dbd43d04c7d4858ec17807d9dd41e58e0" exitCode=0
Oct 31 23:24:22.532671 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.532622    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6685e-7c669d9694-8jd86" event={"ID":"b9eac259-b8e2-4fff-a019-2daf3c17c47b","Type":"ContainerDied","Data":"57d9d967eb5a8fe9fb27f47d3ebf2d1dbd43d04c7d4858ec17807d9dd41e58e0"}
Oct 31 23:24:22.532671 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.532666    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6685e-7c669d9694-8jd86" event={"ID":"b9eac259-b8e2-4fff-a019-2daf3c17c47b","Type":"ContainerDied","Data":"969e6f259c88dff26a8a61754ccecc755d2f029743bcf767860064d14f57b8b3"}
Oct 31 23:24:22.532789 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.532729    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d6685e-7c669d9694-8jd86"
Oct 31 23:24:22.542414 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.542399    4301 scope.go:122] "RemoveContainer" containerID="49034d96749eaaea772cacd57f65025100f34fae047aeb6bfec1a05d795eb028"
Oct 31 23:24:22.542710 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:24:22.542692    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49034d96749eaaea772cacd57f65025100f34fae047aeb6bfec1a05d795eb028\": container with ID starting with 49034d96749eaaea772cacd57f65025100f34fae047aeb6bfec1a05d795eb028 not found: ID does not exist" containerID="49034d96749eaaea772cacd57f65025100f34fae047aeb6bfec1a05d795eb028"
Oct 31 23:24:22.542789 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.542720    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49034d96749eaaea772cacd57f65025100f34fae047aeb6bfec1a05d795eb028"} err="failed to get container status \"49034d96749eaaea772cacd57f65025100f34fae047aeb6bfec1a05d795eb028\": rpc error: code = NotFound desc = could not find container \"49034d96749eaaea772cacd57f65025100f34fae047aeb6bfec1a05d795eb028\": container with ID starting with 49034d96749eaaea772cacd57f65025100f34fae047aeb6bfec1a05d795eb028 not found: ID does not exist"
Oct 31 23:24:22.542789 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.542742    4301 scope.go:122] "RemoveContainer" containerID="5e73ef3a2e757e8886446d510f728e56d30c75c9780c3bf40494967422e6084b"
Oct 31 23:24:22.554229 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.554209    4301 scope.go:122] "RemoveContainer" containerID="5e73ef3a2e757e8886446d510f728e56d30c75c9780c3bf40494967422e6084b"
Oct 31 23:24:22.554555 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:24:22.554529    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e73ef3a2e757e8886446d510f728e56d30c75c9780c3bf40494967422e6084b\": container with ID starting with 5e73ef3a2e757e8886446d510f728e56d30c75c9780c3bf40494967422e6084b not found: ID does not exist" containerID="5e73ef3a2e757e8886446d510f728e56d30c75c9780c3bf40494967422e6084b"
Oct 31 23:24:22.554631 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.554559    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e73ef3a2e757e8886446d510f728e56d30c75c9780c3bf40494967422e6084b"} err="failed to get container status \"5e73ef3a2e757e8886446d510f728e56d30c75c9780c3bf40494967422e6084b\": rpc error: code = NotFound desc = could not find container \"5e73ef3a2e757e8886446d510f728e56d30c75c9780c3bf40494967422e6084b\": container with ID starting with 5e73ef3a2e757e8886446d510f728e56d30c75c9780c3bf40494967422e6084b not found: ID does not exist"
Oct 31 23:24:22.554631 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.554577    4301 scope.go:122] "RemoveContainer" containerID="57d9d967eb5a8fe9fb27f47d3ebf2d1dbd43d04c7d4858ec17807d9dd41e58e0"
Oct 31 23:24:22.560871 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.560850    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f80dfa-847989bcd7-dcg6p"]
Oct 31 23:24:22.566350 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.566324    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f80dfa-847989bcd7-dcg6p"]
Oct 31 23:24:22.571304 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.571048    4301 scope.go:122] "RemoveContainer" containerID="57d9d967eb5a8fe9fb27f47d3ebf2d1dbd43d04c7d4858ec17807d9dd41e58e0"
Oct 31 23:24:22.571440 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:24:22.571424    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57d9d967eb5a8fe9fb27f47d3ebf2d1dbd43d04c7d4858ec17807d9dd41e58e0\": container with ID starting with 57d9d967eb5a8fe9fb27f47d3ebf2d1dbd43d04c7d4858ec17807d9dd41e58e0 not found: ID does not exist" containerID="57d9d967eb5a8fe9fb27f47d3ebf2d1dbd43d04c7d4858ec17807d9dd41e58e0"
Oct 31 23:24:22.571502 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.571449    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57d9d967eb5a8fe9fb27f47d3ebf2d1dbd43d04c7d4858ec17807d9dd41e58e0"} err="failed to get container status \"57d9d967eb5a8fe9fb27f47d3ebf2d1dbd43d04c7d4858ec17807d9dd41e58e0\": rpc error: code = NotFound desc = could not find container \"57d9d967eb5a8fe9fb27f47d3ebf2d1dbd43d04c7d4858ec17807d9dd41e58e0\": container with ID starting with 57d9d967eb5a8fe9fb27f47d3ebf2d1dbd43d04c7d4858ec17807d9dd41e58e0 not found: ID does not exist"
Oct 31 23:24:22.573762 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.573720    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6685e-7bdfff8567-s9nqv"]
Oct 31 23:24:22.577544 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.577523    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d6685e-7bdfff8567-s9nqv"]
Oct 31 23:24:22.586975 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.586955    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d6685e-7c669d9694-8jd86"]
Oct 31 23:24:22.591161 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:22.591142    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d6685e-7c669d9694-8jd86"]
Oct 31 23:24:23.474387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:23.466815    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35ceb2a4-953d-4719-9d85-e3c8491d1825" path="/var/lib/kubelet/pods/35ceb2a4-953d-4719-9d85-e3c8491d1825/volumes"
Oct 31 23:24:23.474387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:23.467511    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9eac259-b8e2-4fff-a019-2daf3c17c47b" path="/var/lib/kubelet/pods/b9eac259-b8e2-4fff-a019-2daf3c17c47b/volumes"
Oct 31 23:24:23.474387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:23.468069    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faee4004-8214-4ca7-9dc4-da1a18620c17" path="/var/lib/kubelet/pods/faee4004-8214-4ca7-9dc4-da1a18620c17/volumes"
Oct 31 23:24:23.919426 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:23.919393    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bf279c-8d7bdc44b-j7pcc"]
Oct 31 23:24:26.044494 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:26.043129    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"]
Oct 31 23:24:26.044494 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:26.043391    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6" podUID="19145c81-9796-4ca8-92b4-8098589f845e" containerName="istio-proxy" containerID="cri-o://b243d9f2ce515e523a8a9a44b6bb247480f65fd932bb9d12dcd9a9cbce1a0879" gracePeriod=30
Oct 31 23:24:28.802568 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:28.802531    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-270e8c-cfb487cd-xpmx2"]
Oct 31 23:24:28.919889 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:28.919847    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-bf279c-8d7bdc44b-j7pcc" podUID="63efe5e5-d71d-41e1-a359-d8d13471f32b" containerName="payload-processing" containerID="cri-o://0f8773480712a972cad285d95319cb75993ace445fc193bb593478e0fe694751" gracePeriod=25
Oct 31 23:24:29.099887 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:29.099788    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"]
Oct 31 23:24:29.101207 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:29.100349    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7" podUID="3cab9805-e8f6-4028-9418-d522053546db" containerName="istio-proxy" containerID="cri-o://5b3f84930779e3fd10a24319161428699fe81315605730ff462000f88125318f" gracePeriod=30
Oct 31 23:24:29.707379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:29.699500    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bf279c-8d7bdc44b-j7pcc"
Oct 31 23:24:29.744575 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:29.744532    4301 generic.go:356] "Generic (PLEG): container finished" podID="63efe5e5-d71d-41e1-a359-d8d13471f32b" containerID="0f8773480712a972cad285d95319cb75993ace445fc193bb593478e0fe694751" exitCode=0
Oct 31 23:24:29.744784 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:29.744729    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bf279c-8d7bdc44b-j7pcc" event={"ID":"63efe5e5-d71d-41e1-a359-d8d13471f32b","Type":"ContainerDied","Data":"0f8773480712a972cad285d95319cb75993ace445fc193bb593478e0fe694751"}
Oct 31 23:24:29.744852 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:29.744839    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bf279c-8d7bdc44b-j7pcc" event={"ID":"63efe5e5-d71d-41e1-a359-d8d13471f32b","Type":"ContainerDied","Data":"cb413acd712e6a629f359a8bf71deaa0b55533d68f6c7a2396f787815a972f41"}
Oct 31 23:24:29.744911 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:29.744871    4301 scope.go:122] "RemoveContainer" containerID="0f8773480712a972cad285d95319cb75993ace445fc193bb593478e0fe694751"
Oct 31 23:24:29.745081 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:29.745061    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bf279c-8d7bdc44b-j7pcc"
Oct 31 23:24:29.785206 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:29.785180    4301 scope.go:122] "RemoveContainer" containerID="0f8773480712a972cad285d95319cb75993ace445fc193bb593478e0fe694751"
Oct 31 23:24:29.785930 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:24:29.785906    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f8773480712a972cad285d95319cb75993ace445fc193bb593478e0fe694751\": container with ID starting with 0f8773480712a972cad285d95319cb75993ace445fc193bb593478e0fe694751 not found: ID does not exist" containerID="0f8773480712a972cad285d95319cb75993ace445fc193bb593478e0fe694751"
Oct 31 23:24:29.786085 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:29.786058    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f8773480712a972cad285d95319cb75993ace445fc193bb593478e0fe694751"} err="failed to get container status \"0f8773480712a972cad285d95319cb75993ace445fc193bb593478e0fe694751\": rpc error: code = NotFound desc = could not find container \"0f8773480712a972cad285d95319cb75993ace445fc193bb593478e0fe694751\": container with ID starting with 0f8773480712a972cad285d95319cb75993ace445fc193bb593478e0fe694751 not found: ID does not exist"
Oct 31 23:24:29.844011 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:29.843899    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63efe5e5-d71d-41e1-a359-d8d13471f32b-kube-api-access-z8594\" (UniqueName: \"kubernetes.io/projected/63efe5e5-d71d-41e1-a359-d8d13471f32b-kube-api-access-z8594\") pod \"63efe5e5-d71d-41e1-a359-d8d13471f32b\" (UID: \"63efe5e5-d71d-41e1-a359-d8d13471f32b\") "
Oct 31 23:24:29.860615 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:29.860582    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63efe5e5-d71d-41e1-a359-d8d13471f32b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63efe5e5-d71d-41e1-a359-d8d13471f32b-plugins-config-volume\") pod \"63efe5e5-d71d-41e1-a359-d8d13471f32b\" (UID: \"63efe5e5-d71d-41e1-a359-d8d13471f32b\") "
Oct 31 23:24:29.862236 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:29.861188    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63efe5e5-d71d-41e1-a359-d8d13471f32b-plugins-config-volume" pod "63efe5e5-d71d-41e1-a359-d8d13471f32b" (UID: "63efe5e5-d71d-41e1-a359-d8d13471f32b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:24:29.874628 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:29.874580    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63efe5e5-d71d-41e1-a359-d8d13471f32b-kube-api-access-z8594" pod "63efe5e5-d71d-41e1-a359-d8d13471f32b" (UID: "63efe5e5-d71d-41e1-a359-d8d13471f32b"). InnerVolumeSpecName "kube-api-access-z8594". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:29.971590 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:29.971545    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8594\" (UniqueName: \"kubernetes.io/projected/63efe5e5-d71d-41e1-a359-d8d13471f32b-kube-api-access-z8594\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:29.971590 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:29.971574    4301 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63efe5e5-d71d-41e1-a359-d8d13471f32b-plugins-config-volume\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:30.224388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:30.220435    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bf279c-8d7bdc44b-j7pcc"]
Oct 31 23:24:30.242382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:30.237609    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bf279c-8d7bdc44b-j7pcc"]
Oct 31 23:24:31.196844 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.196804    4301 patch_prober.go:29] interesting pod/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 23:24:31.197190 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.196888    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7" podUID="3cab9805-e8f6-4028-9418-d522053546db" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 23:24:31.318580 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.318555    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:24:31.405967 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.405927    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63efe5e5-d71d-41e1-a359-d8d13471f32b" path="/var/lib/kubelet/pods/63efe5e5-d71d-41e1-a359-d8d13471f32b/volumes"
Oct 31 23:24:31.422059 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.422031    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19145c81-9796-4ca8-92b4-8098589f845e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19145c81-9796-4ca8-92b4-8098589f845e-istio-podinfo\") pod \"19145c81-9796-4ca8-92b4-8098589f845e\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") "
Oct 31 23:24:31.422183 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.422159    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-istio-data\") pod \"19145c81-9796-4ca8-92b4-8098589f845e\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") "
Oct 31 23:24:31.422249 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.422208    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19145c81-9796-4ca8-92b4-8098589f845e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19145c81-9796-4ca8-92b4-8098589f845e-istiod-ca-cert\") pod \"19145c81-9796-4ca8-92b4-8098589f845e\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") "
Oct 31 23:24:31.422286 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.422245    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-workload-socket\") pod \"19145c81-9796-4ca8-92b4-8098589f845e\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") "
Oct 31 23:24:31.422370 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.422345    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19145c81-9796-4ca8-92b4-8098589f845e-istio-token\" (UniqueName: \"kubernetes.io/projected/19145c81-9796-4ca8-92b4-8098589f845e-istio-token\") pod \"19145c81-9796-4ca8-92b4-8098589f845e\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") "
Oct 31 23:24:31.422443 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.422429    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-workload-certs\") pod \"19145c81-9796-4ca8-92b4-8098589f845e\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") "
Oct 31 23:24:31.422500 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.422481    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-istio-envoy\") pod \"19145c81-9796-4ca8-92b4-8098589f845e\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") "
Oct 31 23:24:31.422500 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.422490    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-istio-data" pod "19145c81-9796-4ca8-92b4-8098589f845e" (UID: "19145c81-9796-4ca8-92b4-8098589f845e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:31.422617 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.422536    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-credential-socket\") pod \"19145c81-9796-4ca8-92b4-8098589f845e\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") "
Oct 31 23:24:31.422617 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.422562    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19145c81-9796-4ca8-92b4-8098589f845e-kube-api-access-vhd4r\" (UniqueName: \"kubernetes.io/projected/19145c81-9796-4ca8-92b4-8098589f845e-kube-api-access-vhd4r\") pod \"19145c81-9796-4ca8-92b4-8098589f845e\" (UID: \"19145c81-9796-4ca8-92b4-8098589f845e\") "
Oct 31 23:24:31.422617 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.422572    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-workload-socket" pod "19145c81-9796-4ca8-92b4-8098589f845e" (UID: "19145c81-9796-4ca8-92b4-8098589f845e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:31.422763 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.422634    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19145c81-9796-4ca8-92b4-8098589f845e-istiod-ca-cert" pod "19145c81-9796-4ca8-92b4-8098589f845e" (UID: "19145c81-9796-4ca8-92b4-8098589f845e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:24:31.422817 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.422770    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-workload-certs" pod "19145c81-9796-4ca8-92b4-8098589f845e" (UID: "19145c81-9796-4ca8-92b4-8098589f845e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:31.422951 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.422907    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-credential-socket" pod "19145c81-9796-4ca8-92b4-8098589f845e" (UID: "19145c81-9796-4ca8-92b4-8098589f845e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:31.423236 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.423199    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:31.423236 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.423220    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:31.423236 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.423234    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:31.423511 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.423248    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19145c81-9796-4ca8-92b4-8098589f845e-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:31.423511 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.423263    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:31.424700 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.424679    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19145c81-9796-4ca8-92b4-8098589f845e-istio-token" pod "19145c81-9796-4ca8-92b4-8098589f845e" (UID: "19145c81-9796-4ca8-92b4-8098589f845e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:31.424793 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.424752    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19145c81-9796-4ca8-92b4-8098589f845e-istio-podinfo" pod "19145c81-9796-4ca8-92b4-8098589f845e" (UID: "19145c81-9796-4ca8-92b4-8098589f845e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:24:31.424868 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.424853    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19145c81-9796-4ca8-92b4-8098589f845e-kube-api-access-vhd4r" pod "19145c81-9796-4ca8-92b4-8098589f845e" (UID: "19145c81-9796-4ca8-92b4-8098589f845e"). InnerVolumeSpecName "kube-api-access-vhd4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:31.424918 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.424873    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-istio-envoy" pod "19145c81-9796-4ca8-92b4-8098589f845e" (UID: "19145c81-9796-4ca8-92b4-8098589f845e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:31.524270 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.524231    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19145c81-9796-4ca8-92b4-8098589f845e-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:31.524270 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.524265    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19145c81-9796-4ca8-92b4-8098589f845e-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:31.524520 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.524291    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhd4r\" (UniqueName: \"kubernetes.io/projected/19145c81-9796-4ca8-92b4-8098589f845e-kube-api-access-vhd4r\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:31.524520 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.524305    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19145c81-9796-4ca8-92b4-8098589f845e-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:31.761373 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.761272    4301 generic.go:356] "Generic (PLEG): container finished" podID="19145c81-9796-4ca8-92b4-8098589f845e" containerID="b243d9f2ce515e523a8a9a44b6bb247480f65fd932bb9d12dcd9a9cbce1a0879" exitCode=0
Oct 31 23:24:31.761373 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.761326    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"
Oct 31 23:24:31.761571 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.761352    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6" event={"ID":"19145c81-9796-4ca8-92b4-8098589f845e","Type":"ContainerDied","Data":"b243d9f2ce515e523a8a9a44b6bb247480f65fd932bb9d12dcd9a9cbce1a0879"}
Oct 31 23:24:31.761571 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.761446    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6" event={"ID":"19145c81-9796-4ca8-92b4-8098589f845e","Type":"ContainerDied","Data":"e952bec911559e119429b7c007e972c748ac2c7971b422570180d4cdd1c747e7"}
Oct 31 23:24:31.761571 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.761478    4301 scope.go:122] "RemoveContainer" containerID="b243d9f2ce515e523a8a9a44b6bb247480f65fd932bb9d12dcd9a9cbce1a0879"
Oct 31 23:24:31.773138 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.773118    4301 scope.go:122] "RemoveContainer" containerID="3e7307870fa89d74731eb6582dc09ccb516618af4a4ad68d4063f9020d13c385"
Oct 31 23:24:31.782281 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.782259    4301 scope.go:122] "RemoveContainer" containerID="b243d9f2ce515e523a8a9a44b6bb247480f65fd932bb9d12dcd9a9cbce1a0879"
Oct 31 23:24:31.782622 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:24:31.782607    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b243d9f2ce515e523a8a9a44b6bb247480f65fd932bb9d12dcd9a9cbce1a0879\": container with ID starting with b243d9f2ce515e523a8a9a44b6bb247480f65fd932bb9d12dcd9a9cbce1a0879 not found: ID does not exist" containerID="b243d9f2ce515e523a8a9a44b6bb247480f65fd932bb9d12dcd9a9cbce1a0879"
Oct 31 23:24:31.782680 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.782631    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b243d9f2ce515e523a8a9a44b6bb247480f65fd932bb9d12dcd9a9cbce1a0879"} err="failed to get container status \"b243d9f2ce515e523a8a9a44b6bb247480f65fd932bb9d12dcd9a9cbce1a0879\": rpc error: code = NotFound desc = could not find container \"b243d9f2ce515e523a8a9a44b6bb247480f65fd932bb9d12dcd9a9cbce1a0879\": container with ID starting with b243d9f2ce515e523a8a9a44b6bb247480f65fd932bb9d12dcd9a9cbce1a0879 not found: ID does not exist"
Oct 31 23:24:31.782680 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.782647    4301 scope.go:122] "RemoveContainer" containerID="3e7307870fa89d74731eb6582dc09ccb516618af4a4ad68d4063f9020d13c385"
Oct 31 23:24:31.782869 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:24:31.782852    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e7307870fa89d74731eb6582dc09ccb516618af4a4ad68d4063f9020d13c385\": container with ID starting with 3e7307870fa89d74731eb6582dc09ccb516618af4a4ad68d4063f9020d13c385 not found: ID does not exist" containerID="3e7307870fa89d74731eb6582dc09ccb516618af4a4ad68d4063f9020d13c385"
Oct 31 23:24:31.782906 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.782884    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e7307870fa89d74731eb6582dc09ccb516618af4a4ad68d4063f9020d13c385"} err="failed to get container status \"3e7307870fa89d74731eb6582dc09ccb516618af4a4ad68d4063f9020d13c385\": rpc error: code = NotFound desc = could not find container \"3e7307870fa89d74731eb6582dc09ccb516618af4a4ad68d4063f9020d13c385\": container with ID starting with 3e7307870fa89d74731eb6582dc09ccb516618af4a4ad68d4063f9020d13c385 not found: ID does not exist"
Oct 31 23:24:31.788754 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.788731    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"]
Oct 31 23:24:31.793038 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:31.793013    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f80dfa-openshift-default-ff868d8f9-8m8f6"]
Oct 31 23:24:32.546710 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:24:32.546672    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:24:33.404331 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:33.404301    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19145c81-9796-4ca8-92b4-8098589f845e" path="/var/lib/kubelet/pods/19145c81-9796-4ca8-92b4-8098589f845e/volumes"
Oct 31 23:24:33.802921 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:33.802827    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-270e8c-cfb487cd-xpmx2" podUID="59d49565-55d0-4f45-8dd4-64bbe81bad3d" containerName="payload-processing" containerID="cri-o://a31acc4094c020a340ea7a6722a401edb838c6494e8642150f801c096297791b" gracePeriod=25
Oct 31 23:24:34.069432 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.069318    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-270e8c-cfb487cd-xpmx2"
Oct 31 23:24:34.177901 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.177871    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59d49565-55d0-4f45-8dd4-64bbe81bad3d-kube-api-access-95sds\" (UniqueName: \"kubernetes.io/projected/59d49565-55d0-4f45-8dd4-64bbe81bad3d-kube-api-access-95sds\") pod \"59d49565-55d0-4f45-8dd4-64bbe81bad3d\" (UID: \"59d49565-55d0-4f45-8dd4-64bbe81bad3d\") "
Oct 31 23:24:34.178055 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.177942    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59d49565-55d0-4f45-8dd4-64bbe81bad3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59d49565-55d0-4f45-8dd4-64bbe81bad3d-plugins-config-volume\") pod \"59d49565-55d0-4f45-8dd4-64bbe81bad3d\" (UID: \"59d49565-55d0-4f45-8dd4-64bbe81bad3d\") "
Oct 31 23:24:34.178326 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.178305    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59d49565-55d0-4f45-8dd4-64bbe81bad3d-plugins-config-volume" pod "59d49565-55d0-4f45-8dd4-64bbe81bad3d" (UID: "59d49565-55d0-4f45-8dd4-64bbe81bad3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:24:34.178528 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.178514    4301 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59d49565-55d0-4f45-8dd4-64bbe81bad3d-plugins-config-volume\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:34.180053 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.180034    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59d49565-55d0-4f45-8dd4-64bbe81bad3d-kube-api-access-95sds" pod "59d49565-55d0-4f45-8dd4-64bbe81bad3d" (UID: "59d49565-55d0-4f45-8dd4-64bbe81bad3d"). InnerVolumeSpecName "kube-api-access-95sds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:34.281403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.281322    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95sds\" (UniqueName: \"kubernetes.io/projected/59d49565-55d0-4f45-8dd4-64bbe81bad3d-kube-api-access-95sds\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:34.440303 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.440273    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:24:34.585447 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.585412    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-workload-socket\") pod \"3cab9805-e8f6-4028-9418-d522053546db\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") "
Oct 31 23:24:34.585644 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.585570    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-credential-socket\") pod \"3cab9805-e8f6-4028-9418-d522053546db\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") "
Oct 31 23:24:34.585715 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.585698    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-istio-data\") pod \"3cab9805-e8f6-4028-9418-d522053546db\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") "
Oct 31 23:24:34.585757 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.585706    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"]
Oct 31 23:24:34.585757 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.585744    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-workload-socket" pod "3cab9805-e8f6-4028-9418-d522053546db" (UID: "3cab9805-e8f6-4028-9418-d522053546db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:34.585841 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.585773    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3cab9805-e8f6-4028-9418-d522053546db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cab9805-e8f6-4028-9418-d522053546db-istio-podinfo\") pod \"3cab9805-e8f6-4028-9418-d522053546db\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") "
Oct 31 23:24:34.585841 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.585815    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cab9805-e8f6-4028-9418-d522053546db-kube-api-access-2xq92\" (UniqueName: \"kubernetes.io/projected/3cab9805-e8f6-4028-9418-d522053546db-kube-api-access-2xq92\") pod \"3cab9805-e8f6-4028-9418-d522053546db\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") "
Oct 31 23:24:34.585936 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.585881    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-istio-envoy\") pod \"3cab9805-e8f6-4028-9418-d522053546db\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") "
Oct 31 23:24:34.585936 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.585903    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-credential-socket" pod "3cab9805-e8f6-4028-9418-d522053546db" (UID: "3cab9805-e8f6-4028-9418-d522053546db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:34.585936 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.585912    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-workload-certs\") pod \"3cab9805-e8f6-4028-9418-d522053546db\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") "
Oct 31 23:24:34.585936 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.585926    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-istio-data" pod "3cab9805-e8f6-4028-9418-d522053546db" (UID: "3cab9805-e8f6-4028-9418-d522053546db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:34.586145 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.585980    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cab9805-e8f6-4028-9418-d522053546db-istio-token\" (UniqueName: \"kubernetes.io/projected/3cab9805-e8f6-4028-9418-d522053546db-istio-token\") pod \"3cab9805-e8f6-4028-9418-d522053546db\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") "
Oct 31 23:24:34.586145 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.586008    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cab9805-e8f6-4028-9418-d522053546db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cab9805-e8f6-4028-9418-d522053546db-istiod-ca-cert\") pod \"3cab9805-e8f6-4028-9418-d522053546db\" (UID: \"3cab9805-e8f6-4028-9418-d522053546db\") "
Oct 31 23:24:34.586145 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.586109    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-workload-certs" pod "3cab9805-e8f6-4028-9418-d522053546db" (UID: "3cab9805-e8f6-4028-9418-d522053546db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:34.586600 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.586576    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cab9805-e8f6-4028-9418-d522053546db-istiod-ca-cert" pod "3cab9805-e8f6-4028-9418-d522053546db" (UID: "3cab9805-e8f6-4028-9418-d522053546db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:24:34.587422 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.587400    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cab9805-e8f6-4028-9418-d522053546db-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:34.587526 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.587424    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:34.587526 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.587439    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:34.587526 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.587450    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:34.587526 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.587459    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:34.588901 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.588859    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-istio-envoy" pod "3cab9805-e8f6-4028-9418-d522053546db" (UID: "3cab9805-e8f6-4028-9418-d522053546db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:34.589303 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.589276    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3cab9805-e8f6-4028-9418-d522053546db-istio-podinfo" pod "3cab9805-e8f6-4028-9418-d522053546db" (UID: "3cab9805-e8f6-4028-9418-d522053546db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:24:34.589420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.589335    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cab9805-e8f6-4028-9418-d522053546db-kube-api-access-2xq92" pod "3cab9805-e8f6-4028-9418-d522053546db" (UID: "3cab9805-e8f6-4028-9418-d522053546db"). InnerVolumeSpecName "kube-api-access-2xq92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:34.590071 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.590045    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cab9805-e8f6-4028-9418-d522053546db-istio-token" pod "3cab9805-e8f6-4028-9418-d522053546db" (UID: "3cab9805-e8f6-4028-9418-d522053546db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:34.689122 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.689085    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cab9805-e8f6-4028-9418-d522053546db-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:34.689122 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.689118    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xq92\" (UniqueName: \"kubernetes.io/projected/3cab9805-e8f6-4028-9418-d522053546db-kube-api-access-2xq92\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:34.689330 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.689135    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cab9805-e8f6-4028-9418-d522053546db-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:34.689330 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.689145    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cab9805-e8f6-4028-9418-d522053546db-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:34.785502 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.785436    4301 generic.go:356] "Generic (PLEG): container finished" podID="59d49565-55d0-4f45-8dd4-64bbe81bad3d" containerID="a31acc4094c020a340ea7a6722a401edb838c6494e8642150f801c096297791b" exitCode=0
Oct 31 23:24:34.785502 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.785490    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-270e8c-cfb487cd-xpmx2"
Oct 31 23:24:34.785761 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.785512    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-270e8c-cfb487cd-xpmx2" event={"ID":"59d49565-55d0-4f45-8dd4-64bbe81bad3d","Type":"ContainerDied","Data":"a31acc4094c020a340ea7a6722a401edb838c6494e8642150f801c096297791b"}
Oct 31 23:24:34.785761 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.785573    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-270e8c-cfb487cd-xpmx2" event={"ID":"59d49565-55d0-4f45-8dd4-64bbe81bad3d","Type":"ContainerDied","Data":"6d77397aa88a4e3e928b4debb4449ad9b44cca457ab8ee869d8203913d305e86"}
Oct 31 23:24:34.785761 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.785614    4301 scope.go:122] "RemoveContainer" containerID="a31acc4094c020a340ea7a6722a401edb838c6494e8642150f801c096297791b"
Oct 31 23:24:34.787298 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.787267    4301 generic.go:356] "Generic (PLEG): container finished" podID="3cab9805-e8f6-4028-9418-d522053546db" containerID="5b3f84930779e3fd10a24319161428699fe81315605730ff462000f88125318f" exitCode=0
Oct 31 23:24:34.787488 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.787310    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"
Oct 31 23:24:34.787488 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.787327    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7" event={"ID":"3cab9805-e8f6-4028-9418-d522053546db","Type":"ContainerDied","Data":"5b3f84930779e3fd10a24319161428699fe81315605730ff462000f88125318f"}
Oct 31 23:24:34.787488 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.787388    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7" event={"ID":"3cab9805-e8f6-4028-9418-d522053546db","Type":"ContainerDied","Data":"90b3d128e8862db51763609c52d7aaa70fe38f6e109d16997514ddd0effc11c3"}
Oct 31 23:24:34.798341 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.798233    4301 scope.go:122] "RemoveContainer" containerID="a31acc4094c020a340ea7a6722a401edb838c6494e8642150f801c096297791b"
Oct 31 23:24:34.798868 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:24:34.798843    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a31acc4094c020a340ea7a6722a401edb838c6494e8642150f801c096297791b\": container with ID starting with a31acc4094c020a340ea7a6722a401edb838c6494e8642150f801c096297791b not found: ID does not exist" containerID="a31acc4094c020a340ea7a6722a401edb838c6494e8642150f801c096297791b"
Oct 31 23:24:34.798960 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.798882    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31acc4094c020a340ea7a6722a401edb838c6494e8642150f801c096297791b"} err="failed to get container status \"a31acc4094c020a340ea7a6722a401edb838c6494e8642150f801c096297791b\": rpc error: code = NotFound desc = could not find container \"a31acc4094c020a340ea7a6722a401edb838c6494e8642150f801c096297791b\": container with ID starting with a31acc4094c020a340ea7a6722a401edb838c6494e8642150f801c096297791b not found: ID does not exist"
Oct 31 23:24:34.798960 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.798904    4301 scope.go:122] "RemoveContainer" containerID="5b3f84930779e3fd10a24319161428699fe81315605730ff462000f88125318f"
Oct 31 23:24:34.809181 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.809011    4301 scope.go:122] "RemoveContainer" containerID="5b3f84930779e3fd10a24319161428699fe81315605730ff462000f88125318f"
Oct 31 23:24:34.809485 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:24:34.809329    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b3f84930779e3fd10a24319161428699fe81315605730ff462000f88125318f\": container with ID starting with 5b3f84930779e3fd10a24319161428699fe81315605730ff462000f88125318f not found: ID does not exist" containerID="5b3f84930779e3fd10a24319161428699fe81315605730ff462000f88125318f"
Oct 31 23:24:34.809485 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.809352    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b3f84930779e3fd10a24319161428699fe81315605730ff462000f88125318f"} err="failed to get container status \"5b3f84930779e3fd10a24319161428699fe81315605730ff462000f88125318f\": rpc error: code = NotFound desc = could not find container \"5b3f84930779e3fd10a24319161428699fe81315605730ff462000f88125318f\": container with ID starting with 5b3f84930779e3fd10a24319161428699fe81315605730ff462000f88125318f not found: ID does not exist"
Oct 31 23:24:34.810922 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.810903    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-270e8c-cfb487cd-xpmx2"]
Oct 31 23:24:34.818977 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.818947    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-270e8c-cfb487cd-xpmx2"]
Oct 31 23:24:34.828860 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.828791    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"]
Oct 31 23:24:34.834019 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:34.833992    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-d5mt7"]
Oct 31 23:24:35.415090 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:35.411455    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cab9805-e8f6-4028-9418-d522053546db" path="/var/lib/kubelet/pods/3cab9805-e8f6-4028-9418-d522053546db/volumes"
Oct 31 23:24:35.415090 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:35.412136    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59d49565-55d0-4f45-8dd4-64bbe81bad3d" path="/var/lib/kubelet/pods/59d49565-55d0-4f45-8dd4-64bbe81bad3d/volumes"
Oct 31 23:24:35.422661 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:35.422630    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"]
Oct 31 23:24:35.440837 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:35.440383    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"]
Oct 31 23:24:35.454826 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:35.448605    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"]
Oct 31 23:24:35.461813 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:35.461778    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"]
Oct 31 23:24:39.890255 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:39.890212    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7d7eed-7c97f4ff9f-whnrt"]
Oct 31 23:24:41.658192 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:41.658140    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"]
Oct 31 23:24:41.658699 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:41.658415    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn" podUID="84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3" containerName="istio-proxy" containerID="cri-o://2c6dbfc4df937412f010f1576feb072186ded77215d6407db82849e339eb2b97" gracePeriod=30
Oct 31 23:24:41.661685 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:41.661658    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"]
Oct 31 23:24:41.661889 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:41.661854    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954" podUID="929c3031-62b4-4587-9f53-75de251130c8" containerName="istio-proxy" containerID="cri-o://af1f952860c2f21a81872c77b77106eb28a8ef2e43e68dbdd858ba9bbb9d0b3f" gracePeriod=30
Oct 31 23:24:41.794411 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:41.793025    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s9nl_1f0042ee-6258-4aa8-af48-7568f2f2e8a1/ovn-acl-logging/0.log"
Oct 31 23:24:41.796712 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:41.796684    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s9nl_1f0042ee-6258-4aa8-af48-7568f2f2e8a1/ovn-acl-logging/0.log"
Oct 31 23:24:43.521387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:43.520105    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"]
Oct 31 23:24:43.521387 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:43.520334    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2" podUID="bba885b3-4cbf-4677-bef9-94f492ca8794" containerName="istio-proxy" containerID="cri-o://004c101df93af2590ce5f0f449ca6129709bb62dbf8db2245e80369b76c4b0e0" gracePeriod=30
Oct 31 23:24:43.540166 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:43.540133    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"]
Oct 31 23:24:43.540746 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:43.540374    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p" podUID="394e1575-7171-45a2-8d19-ef551eaad0ea" containerName="istio-proxy" containerID="cri-o://b17227da4c1379614a3ac17d6f7bc0e0c17b0b4b9f4fae005652472059b2797f" gracePeriod=30
Oct 31 23:24:43.545331 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:43.545307    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"]
Oct 31 23:24:43.545555 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:43.545523    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf" podUID="2b5f9bbb-897d-44dc-9eb8-5887091b4d64" containerName="istio-proxy" containerID="cri-o://c46aff3eccf6248c32249539e8a9da8fd01a605fcafdda31143efae0804002ea" gracePeriod=30
Oct 31 23:24:44.628658 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.628624    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de6d2120-x6jhx"]
Oct 31 23:24:44.629351 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629330    4301 cpu_manager.go:414] "containerMap: removing container" podUID="faee4004-8214-4ca7-9dc4-da1a18620c17" containerName="payload-pre-processing"
Oct 31 23:24:44.629474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629353    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faee4004-8214-4ca7-9dc4-da1a18620c17" containerName="payload-pre-processing"
Oct 31 23:24:44.629474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629389    4301 cpu_manager.go:414] "containerMap: removing container" podUID="b9eac259-b8e2-4fff-a019-2daf3c17c47b" containerName="payload-processing"
Oct 31 23:24:44.629474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629396    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9eac259-b8e2-4fff-a019-2daf3c17c47b" containerName="payload-processing"
Oct 31 23:24:44.629474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629424    4301 cpu_manager.go:414] "containerMap: removing container" podUID="19145c81-9796-4ca8-92b4-8098589f845e" containerName="istio-proxy"
Oct 31 23:24:44.629474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629432    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19145c81-9796-4ca8-92b4-8098589f845e" containerName="istio-proxy"
Oct 31 23:24:44.629474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629445    4301 cpu_manager.go:414] "containerMap: removing container" podUID="f8595a54-27b2-43a5-9e7d-f7d0fd41c523" containerName="istio-proxy"
Oct 31 23:24:44.629474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629452    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8595a54-27b2-43a5-9e7d-f7d0fd41c523" containerName="istio-proxy"
Oct 31 23:24:44.629474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629472    4301 cpu_manager.go:414] "containerMap: removing container" podUID="d05b75f2-18af-4d86-99d9-a51d2d23c3b5" containerName="istio-proxy"
Oct 31 23:24:44.629907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629479    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d05b75f2-18af-4d86-99d9-a51d2d23c3b5" containerName="istio-proxy"
Oct 31 23:24:44.629907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629489    4301 cpu_manager.go:414] "containerMap: removing container" podUID="34836e36-7a9c-4072-a555-970e2eb17c2b" containerName="istio-proxy"
Oct 31 23:24:44.629907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629495    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34836e36-7a9c-4072-a555-970e2eb17c2b" containerName="istio-proxy"
Oct 31 23:24:44.629907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629520    4301 cpu_manager.go:414] "containerMap: removing container" podUID="46881476-3103-4ee9-9b90-ec0ba9f30c3e" containerName="istio-proxy"
Oct 31 23:24:44.629907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629527    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46881476-3103-4ee9-9b90-ec0ba9f30c3e" containerName="istio-proxy"
Oct 31 23:24:44.629907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629540    4301 cpu_manager.go:414] "containerMap: removing container" podUID="0a0fbeac-3a35-47cc-89fe-dcd510841e18" containerName="revoke-keys"
Oct 31 23:24:44.629907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629547    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a0fbeac-3a35-47cc-89fe-dcd510841e18" containerName="revoke-keys"
Oct 31 23:24:44.629907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629571    4301 cpu_manager.go:414] "containerMap: removing container" podUID="d05b75f2-18af-4d86-99d9-a51d2d23c3b5" containerName="istio-proxy"
Oct 31 23:24:44.629907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629578    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d05b75f2-18af-4d86-99d9-a51d2d23c3b5" containerName="istio-proxy"
Oct 31 23:24:44.629907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629589    4301 cpu_manager.go:414] "containerMap: removing container" podUID="3cab9805-e8f6-4028-9418-d522053546db" containerName="istio-proxy"
Oct 31 23:24:44.629907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629597    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cab9805-e8f6-4028-9418-d522053546db" containerName="istio-proxy"
Oct 31 23:24:44.629907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629620    4301 cpu_manager.go:414] "containerMap: removing container" podUID="59d49565-55d0-4f45-8dd4-64bbe81bad3d" containerName="payload-processing"
Oct 31 23:24:44.629907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629627    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59d49565-55d0-4f45-8dd4-64bbe81bad3d" containerName="payload-processing"
Oct 31 23:24:44.629907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629638    4301 cpu_manager.go:414] "containerMap: removing container" podUID="35ceb2a4-953d-4719-9d85-e3c8491d1825" containerName="payload-pre-processing"
Oct 31 23:24:44.629907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629646    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35ceb2a4-953d-4719-9d85-e3c8491d1825" containerName="payload-pre-processing"
Oct 31 23:24:44.629907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629662    4301 cpu_manager.go:414] "containerMap: removing container" podUID="8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2" containerName="istio-proxy"
Oct 31 23:24:44.629907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629668    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2" containerName="istio-proxy"
Oct 31 23:24:44.629907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629680    4301 cpu_manager.go:414] "containerMap: removing container" podUID="b1a1157d-88ea-448f-931d-9528c5df5684" containerName="istio-proxy"
Oct 31 23:24:44.629907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629686    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1a1157d-88ea-448f-931d-9528c5df5684" containerName="istio-proxy"
Oct 31 23:24:44.629907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629699    4301 cpu_manager.go:414] "containerMap: removing container" podUID="1fd21e5d-47da-405b-9724-84cafcd106d2" containerName="istio-proxy"
Oct 31 23:24:44.629907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629706    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fd21e5d-47da-405b-9724-84cafcd106d2" containerName="istio-proxy"
Oct 31 23:24:44.629907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629718    4301 cpu_manager.go:414] "containerMap: removing container" podUID="46881476-3103-4ee9-9b90-ec0ba9f30c3e" containerName="istio-proxy"
Oct 31 23:24:44.629907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629724    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46881476-3103-4ee9-9b90-ec0ba9f30c3e" containerName="istio-proxy"
Oct 31 23:24:44.629907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629735    4301 cpu_manager.go:414] "containerMap: removing container" podUID="63efe5e5-d71d-41e1-a359-d8d13471f32b" containerName="payload-processing"
Oct 31 23:24:44.629907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629742    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63efe5e5-d71d-41e1-a359-d8d13471f32b" containerName="payload-processing"
Oct 31 23:24:44.629907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629751    4301 cpu_manager.go:414] "containerMap: removing container" podUID="e5c2b8fb-331d-4df2-ad30-13549d8a660c" containerName="istio-proxy"
Oct 31 23:24:44.629907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629758    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5c2b8fb-331d-4df2-ad30-13549d8a660c" containerName="istio-proxy"
Oct 31 23:24:44.629907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629886    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5c2b8fb-331d-4df2-ad30-13549d8a660c" containerName="istio-proxy"
Oct 31 23:24:44.629907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629904    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="faee4004-8214-4ca7-9dc4-da1a18620c17" containerName="payload-pre-processing"
Oct 31 23:24:44.629907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629914    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9eac259-b8e2-4fff-a019-2daf3c17c47b" containerName="payload-processing"
Oct 31 23:24:44.631403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629927    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="19145c81-9796-4ca8-92b4-8098589f845e" containerName="istio-proxy"
Oct 31 23:24:44.631403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629936    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8595a54-27b2-43a5-9e7d-f7d0fd41c523" containerName="istio-proxy"
Oct 31 23:24:44.631403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629946    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="d05b75f2-18af-4d86-99d9-a51d2d23c3b5" containerName="istio-proxy"
Oct 31 23:24:44.631403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629954    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="34836e36-7a9c-4072-a555-970e2eb17c2b" containerName="istio-proxy"
Oct 31 23:24:44.631403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629964    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="46881476-3103-4ee9-9b90-ec0ba9f30c3e" containerName="istio-proxy"
Oct 31 23:24:44.631403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629972    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a0fbeac-3a35-47cc-89fe-dcd510841e18" containerName="revoke-keys"
Oct 31 23:24:44.631403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629981    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="d05b75f2-18af-4d86-99d9-a51d2d23c3b5" containerName="istio-proxy"
Oct 31 23:24:44.631403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629989    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cab9805-e8f6-4028-9418-d522053546db" containerName="istio-proxy"
Oct 31 23:24:44.631403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.629998    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="59d49565-55d0-4f45-8dd4-64bbe81bad3d" containerName="payload-processing"
Oct 31 23:24:44.631403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.630009    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="35ceb2a4-953d-4719-9d85-e3c8491d1825" containerName="payload-pre-processing"
Oct 31 23:24:44.631403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.630019    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1a1157d-88ea-448f-931d-9528c5df5684" containerName="istio-proxy"
Oct 31 23:24:44.631403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.630028    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d5e2ad8-17f8-412f-b2c2-24b93d77d1a2" containerName="istio-proxy"
Oct 31 23:24:44.631403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.630037    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fd21e5d-47da-405b-9724-84cafcd106d2" containerName="istio-proxy"
Oct 31 23:24:44.631403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.630045    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="63efe5e5-d71d-41e1-a359-d8d13471f32b" containerName="payload-processing"
Oct 31 23:24:44.636456 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.636434    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6d2120-x6jhx"
Oct 31 23:24:44.644351 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.644317    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 23:24:44.647273 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.647037    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de6d2120-x6jhx"]
Oct 31 23:24:44.648315 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.647784    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\""
Oct 31 23:24:44.693808 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.693772    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft9q5\" (UniqueName: \"kubernetes.io/projected/9760fd20-c563-4f5a-be5a-eeca1a7898a3-kube-api-access-ft9q5\") pod \"maas-api-revoke-sub-de6d2120-x6jhx\" (UID: \"9760fd20-c563-4f5a-be5a-eeca1a7898a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6d2120-x6jhx"
Oct 31 23:24:44.693972 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.693874    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9760fd20-c563-4f5a-be5a-eeca1a7898a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de6d2120-x6jhx\" (UID: \"9760fd20-c563-4f5a-be5a-eeca1a7898a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6d2120-x6jhx"
Oct 31 23:24:44.693972 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.693920    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9760fd20-c563-4f5a-be5a-eeca1a7898a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de6d2120-x6jhx\" (UID: \"9760fd20-c563-4f5a-be5a-eeca1a7898a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6d2120-x6jhx"
Oct 31 23:24:44.795461 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.795425    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft9q5\" (UniqueName: \"kubernetes.io/projected/9760fd20-c563-4f5a-be5a-eeca1a7898a3-kube-api-access-ft9q5\") pod \"maas-api-revoke-sub-de6d2120-x6jhx\" (UID: \"9760fd20-c563-4f5a-be5a-eeca1a7898a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6d2120-x6jhx"
Oct 31 23:24:44.795630 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.795497    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9760fd20-c563-4f5a-be5a-eeca1a7898a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de6d2120-x6jhx\" (UID: \"9760fd20-c563-4f5a-be5a-eeca1a7898a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6d2120-x6jhx"
Oct 31 23:24:44.795630 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.795542    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9760fd20-c563-4f5a-be5a-eeca1a7898a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de6d2120-x6jhx\" (UID: \"9760fd20-c563-4f5a-be5a-eeca1a7898a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6d2120-x6jhx"
Oct 31 23:24:44.796170 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.796142    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9760fd20-c563-4f5a-be5a-eeca1a7898a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de6d2120-x6jhx\" (UID: \"9760fd20-c563-4f5a-be5a-eeca1a7898a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6d2120-x6jhx"
Oct 31 23:24:44.797931 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.797913    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9760fd20-c563-4f5a-be5a-eeca1a7898a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de6d2120-x6jhx\" (UID: \"9760fd20-c563-4f5a-be5a-eeca1a7898a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6d2120-x6jhx"
Oct 31 23:24:44.815108 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.815087    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft9q5\" (UniqueName: \"kubernetes.io/projected/9760fd20-c563-4f5a-be5a-eeca1a7898a3-kube-api-access-ft9q5\") pod \"maas-api-revoke-sub-de6d2120-x6jhx\" (UID: \"9760fd20-c563-4f5a-be5a-eeca1a7898a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6d2120-x6jhx"
Oct 31 23:24:44.890713 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.890633    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d7eed-7c97f4ff9f-whnrt" podUID="a3bc4bb3-e967-4fe3-a6b9-71377f8f6047" containerName="payload-pre-processing" containerID="cri-o://5de4cae8fa83ea1bedeac765e1e1ae9941651750c3419eadbef02ad843453b20" gracePeriod=25
Oct 31 23:24:44.960001 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:44.959959    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6d2120-x6jhx"
Oct 31 23:24:45.100280 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:45.099913    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:24:45.369331 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:45.369141    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de6d2120-x6jhx"]
Oct 31 23:24:45.373394 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:24:45.373001    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9760fd20_c563_4f5a_be5a_eeca1a7898a3.slice/crio-113b2ab6036a280661d382d8d6150bafd74cfcb212a128dd01ec68ea42bdf3fa WatchSource:0}: Error finding container 113b2ab6036a280661d382d8d6150bafd74cfcb212a128dd01ec68ea42bdf3fa: Status 404 returned error can't find the container with id 113b2ab6036a280661d382d8d6150bafd74cfcb212a128dd01ec68ea42bdf3fa
Oct 31 23:24:45.528156 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:45.528118    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"]
Oct 31 23:24:45.528429 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:45.528386    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds" podUID="e9de1723-3c15-4c53-8080-26e27ac118e4" containerName="istio-proxy" containerID="cri-o://192f6f07478d7ec10ff749b7697be9c1ba196660d01b7e03fb5f05cefc0cff0b" gracePeriod=30
Oct 31 23:24:46.332457 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:46.330448    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6d2120-x6jhx" event={"ID":"9760fd20-c563-4f5a-be5a-eeca1a7898a3","Type":"ContainerStarted","Data":"a79446064d779a1cf4e8f95ea9a55920862631b3776a212e2cde8c78baa1fb78"}
Oct 31 23:24:46.332457 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:46.330544    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6d2120-x6jhx" event={"ID":"9760fd20-c563-4f5a-be5a-eeca1a7898a3","Type":"ContainerStarted","Data":"113b2ab6036a280661d382d8d6150bafd74cfcb212a128dd01ec68ea42bdf3fa"}
Oct 31 23:24:46.348969 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:46.348913    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6d2120-x6jhx" podStartSLOduration=2.348894486 podStartE2EDuration="2.348894486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:24:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:24:46.347018643 +0000 UTC m=+2105.094208148" watchObservedRunningTime="2026-10-31 23:24:46.348894486 +0000 UTC m=+2105.096083993"
Oct 31 23:24:46.452246 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:46.452214    4301 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 23:24:46.452456 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:46.452264    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn" podUID="84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 23:24:47.016532 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.016505    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:24:47.062824 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.062795    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:24:47.122221 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.122184    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-workload-certs\") pod \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") "
Oct 31 23:24:47.122455 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.122256    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-credential-socket\") pod \"929c3031-62b4-4587-9f53-75de251130c8\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") "
Oct 31 23:24:47.122455 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.122298    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-workload-socket\") pod \"929c3031-62b4-4587-9f53-75de251130c8\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") "
Oct 31 23:24:47.122455 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.122326    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istio-podinfo\") pod \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") "
Oct 31 23:24:47.122455 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.122352    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-workload-certs\") pod \"929c3031-62b4-4587-9f53-75de251130c8\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") "
Oct 31 23:24:47.122455 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.122436    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-istio-data\") pod \"929c3031-62b4-4587-9f53-75de251130c8\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") "
Oct 31 23:24:47.122759 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.122471    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-kube-api-access-pmr8j\" (UniqueName: \"kubernetes.io/projected/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-kube-api-access-pmr8j\") pod \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") "
Oct 31 23:24:47.122759 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.122521    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istiod-ca-cert\") pod \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") "
Oct 31 23:24:47.122759 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.122588    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-istio-envoy\") pod \"929c3031-62b4-4587-9f53-75de251130c8\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") "
Oct 31 23:24:47.122759 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.122598    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-workload-certs" pod "84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3" (UID: "84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:47.122759 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.122624    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/929c3031-62b4-4587-9f53-75de251130c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/929c3031-62b4-4587-9f53-75de251130c8-istio-podinfo\") pod \"929c3031-62b4-4587-9f53-75de251130c8\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") "
Oct 31 23:24:47.122759 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.122650    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/929c3031-62b4-4587-9f53-75de251130c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/929c3031-62b4-4587-9f53-75de251130c8-istiod-ca-cert\") pod \"929c3031-62b4-4587-9f53-75de251130c8\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") "
Oct 31 23:24:47.122759 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.122647    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-credential-socket" pod "929c3031-62b4-4587-9f53-75de251130c8" (UID: "929c3031-62b4-4587-9f53-75de251130c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:47.122759 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.122676    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-workload-socket" pod "929c3031-62b4-4587-9f53-75de251130c8" (UID: "929c3031-62b4-4587-9f53-75de251130c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:47.122759 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.122697    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/929c3031-62b4-4587-9f53-75de251130c8-istio-token\" (UniqueName: \"kubernetes.io/projected/929c3031-62b4-4587-9f53-75de251130c8-istio-token\") pod \"929c3031-62b4-4587-9f53-75de251130c8\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") "
Oct 31 23:24:47.122759 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.122764    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istio-envoy\") pod \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") "
Oct 31 23:24:47.123264 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.122857    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-workload-socket\") pod \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") "
Oct 31 23:24:47.123264 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.122922    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/929c3031-62b4-4587-9f53-75de251130c8-kube-api-access-mc424\" (UniqueName: \"kubernetes.io/projected/929c3031-62b4-4587-9f53-75de251130c8-kube-api-access-mc424\") pod \"929c3031-62b4-4587-9f53-75de251130c8\" (UID: \"929c3031-62b4-4587-9f53-75de251130c8\") "
Oct 31 23:24:47.123264 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.122959    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istio-data\") pod \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") "
Oct 31 23:24:47.123264 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.122996    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istio-token\" (UniqueName: \"kubernetes.io/projected/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istio-token\") pod \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") "
Oct 31 23:24:47.123264 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.123029    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-credential-socket\") pod \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\" (UID: \"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3\") "
Oct 31 23:24:47.123264 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.123150    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/929c3031-62b4-4587-9f53-75de251130c8-istiod-ca-cert" pod "929c3031-62b4-4587-9f53-75de251130c8" (UID: "929c3031-62b4-4587-9f53-75de251130c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:24:47.123614 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.123419    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-workload-certs" pod "929c3031-62b4-4587-9f53-75de251130c8" (UID: "929c3031-62b4-4587-9f53-75de251130c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:47.123614 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.123593    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-credential-socket" pod "84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3" (UID: "84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:47.123804 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.123746    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-istio-data" pod "929c3031-62b4-4587-9f53-75de251130c8" (UID: "929c3031-62b4-4587-9f53-75de251130c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:47.123804 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.123795    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/929c3031-62b4-4587-9f53-75de251130c8-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:47.124047 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.123812    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:47.124047 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.123827    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:47.124047 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.123840    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:47.124047 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.123857    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:47.124047 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.123869    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:47.124305 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.124058    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istiod-ca-cert" pod "84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3" (UID: "84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:24:47.125423 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.125391    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istio-data" pod "84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3" (UID: "84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:47.126118 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.125828    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-kube-api-access-pmr8j" pod "84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3" (UID: "84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3"). InnerVolumeSpecName "kube-api-access-pmr8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:47.126118 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.126051    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-workload-socket" pod "84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3" (UID: "84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:47.126251 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.126185    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/929c3031-62b4-4587-9f53-75de251130c8-kube-api-access-mc424" pod "929c3031-62b4-4587-9f53-75de251130c8" (UID: "929c3031-62b4-4587-9f53-75de251130c8"). InnerVolumeSpecName "kube-api-access-mc424". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:47.126780 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.126748    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/929c3031-62b4-4587-9f53-75de251130c8-istio-podinfo" pod "929c3031-62b4-4587-9f53-75de251130c8" (UID: "929c3031-62b4-4587-9f53-75de251130c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:24:47.127188 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.127157    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istio-podinfo" pod "84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3" (UID: "84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:24:47.127776 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.127752    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/929c3031-62b4-4587-9f53-75de251130c8-istio-token" pod "929c3031-62b4-4587-9f53-75de251130c8" (UID: "929c3031-62b4-4587-9f53-75de251130c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:47.127928 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.127909    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-istio-envoy" pod "929c3031-62b4-4587-9f53-75de251130c8" (UID: "929c3031-62b4-4587-9f53-75de251130c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:47.132789 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.132767    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istio-envoy" pod "84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3" (UID: "84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:47.132902 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.132850    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istio-token" pod "84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3" (UID: "84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:47.225272 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.225185    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc424\" (UniqueName: \"kubernetes.io/projected/929c3031-62b4-4587-9f53-75de251130c8-kube-api-access-mc424\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:47.225272 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.225217    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:47.225272 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.225228    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:47.225272 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.225236    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:47.225272 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.225245    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:47.225272 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.225254    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmr8j\" (UniqueName: \"kubernetes.io/projected/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-kube-api-access-pmr8j\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:47.225272 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.225265    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:47.225272 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.225278    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/929c3031-62b4-4587-9f53-75de251130c8-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:47.225733 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.225287    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/929c3031-62b4-4587-9f53-75de251130c8-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:47.225733 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.225306    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/929c3031-62b4-4587-9f53-75de251130c8-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:47.225733 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.225315    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:47.225733 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.225323    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:47.338576 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.338543    4301 generic.go:356] "Generic (PLEG): container finished" podID="9760fd20-c563-4f5a-be5a-eeca1a7898a3" containerID="a79446064d779a1cf4e8f95ea9a55920862631b3776a212e2cde8c78baa1fb78" exitCode=0
Oct 31 23:24:47.339052 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.338633    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6d2120-x6jhx" event={"ID":"9760fd20-c563-4f5a-be5a-eeca1a7898a3","Type":"ContainerDied","Data":"a79446064d779a1cf4e8f95ea9a55920862631b3776a212e2cde8c78baa1fb78"}
Oct 31 23:24:47.339992 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.339967    4301 generic.go:356] "Generic (PLEG): container finished" podID="84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3" containerID="2c6dbfc4df937412f010f1576feb072186ded77215d6407db82849e339eb2b97" exitCode=0
Oct 31 23:24:47.340137 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.340023    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"
Oct 31 23:24:47.340137 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.340043    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn" event={"ID":"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3","Type":"ContainerDied","Data":"2c6dbfc4df937412f010f1576feb072186ded77215d6407db82849e339eb2b97"}
Oct 31 23:24:47.340137 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.340073    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn" event={"ID":"84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3","Type":"ContainerDied","Data":"2ed2e7fbd8636b6d866b9d8d947c4d733b492956b208723aa26662490b22e8ac"}
Oct 31 23:24:47.340137 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.340100    4301 scope.go:122] "RemoveContainer" containerID="2c6dbfc4df937412f010f1576feb072186ded77215d6407db82849e339eb2b97"
Oct 31 23:24:47.341570 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.341544    4301 generic.go:356] "Generic (PLEG): container finished" podID="929c3031-62b4-4587-9f53-75de251130c8" containerID="af1f952860c2f21a81872c77b77106eb28a8ef2e43e68dbdd858ba9bbb9d0b3f" exitCode=0
Oct 31 23:24:47.341674 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.341568    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954" event={"ID":"929c3031-62b4-4587-9f53-75de251130c8","Type":"ContainerDied","Data":"af1f952860c2f21a81872c77b77106eb28a8ef2e43e68dbdd858ba9bbb9d0b3f"}
Oct 31 23:24:47.341674 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.341597    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"
Oct 31 23:24:47.341674 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.341633    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954" event={"ID":"929c3031-62b4-4587-9f53-75de251130c8","Type":"ContainerDied","Data":"1f376b976a92e80502163d4a043c1cbd6b9d5306ffefaf56db2d307b2639099a"}
Oct 31 23:24:47.351735 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.351714    4301 scope.go:122] "RemoveContainer" containerID="2c6dbfc4df937412f010f1576feb072186ded77215d6407db82849e339eb2b97"
Oct 31 23:24:47.352045 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:24:47.352026    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c6dbfc4df937412f010f1576feb072186ded77215d6407db82849e339eb2b97\": container with ID starting with 2c6dbfc4df937412f010f1576feb072186ded77215d6407db82849e339eb2b97 not found: ID does not exist" containerID="2c6dbfc4df937412f010f1576feb072186ded77215d6407db82849e339eb2b97"
Oct 31 23:24:47.352109 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.352054    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c6dbfc4df937412f010f1576feb072186ded77215d6407db82849e339eb2b97"} err="failed to get container status \"2c6dbfc4df937412f010f1576feb072186ded77215d6407db82849e339eb2b97\": rpc error: code = NotFound desc = could not find container \"2c6dbfc4df937412f010f1576feb072186ded77215d6407db82849e339eb2b97\": container with ID starting with 2c6dbfc4df937412f010f1576feb072186ded77215d6407db82849e339eb2b97 not found: ID does not exist"
Oct 31 23:24:47.352109 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.352070    4301 scope.go:122] "RemoveContainer" containerID="af1f952860c2f21a81872c77b77106eb28a8ef2e43e68dbdd858ba9bbb9d0b3f"
Oct 31 23:24:47.361630 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.361601    4301 scope.go:122] "RemoveContainer" containerID="af1f952860c2f21a81872c77b77106eb28a8ef2e43e68dbdd858ba9bbb9d0b3f"
Oct 31 23:24:47.361911 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:24:47.361891    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af1f952860c2f21a81872c77b77106eb28a8ef2e43e68dbdd858ba9bbb9d0b3f\": container with ID starting with af1f952860c2f21a81872c77b77106eb28a8ef2e43e68dbdd858ba9bbb9d0b3f not found: ID does not exist" containerID="af1f952860c2f21a81872c77b77106eb28a8ef2e43e68dbdd858ba9bbb9d0b3f"
Oct 31 23:24:47.361961 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.361917    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af1f952860c2f21a81872c77b77106eb28a8ef2e43e68dbdd858ba9bbb9d0b3f"} err="failed to get container status \"af1f952860c2f21a81872c77b77106eb28a8ef2e43e68dbdd858ba9bbb9d0b3f\": rpc error: code = NotFound desc = could not find container \"af1f952860c2f21a81872c77b77106eb28a8ef2e43e68dbdd858ba9bbb9d0b3f\": container with ID starting with af1f952860c2f21a81872c77b77106eb28a8ef2e43e68dbdd858ba9bbb9d0b3f not found: ID does not exist"
Oct 31 23:24:47.370831 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.370808    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"]
Oct 31 23:24:47.374972 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.374950    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-gm954"]
Oct 31 23:24:47.385198 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.385175    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"]
Oct 31 23:24:47.389982 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.389961    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-8cscn"]
Oct 31 23:24:47.404235 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.404212    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3" path="/var/lib/kubelet/pods/84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3/volumes"
Oct 31 23:24:47.404790 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:47.404776    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="929c3031-62b4-4587-9f53-75de251130c8" path="/var/lib/kubelet/pods/929c3031-62b4-4587-9f53-75de251130c8/volumes"
Oct 31 23:24:48.516589 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.516561    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6d2120-x6jhx"
Oct 31 23:24:48.654383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.654336    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9760fd20-c563-4f5a-be5a-eeca1a7898a3-kube-api-access-ft9q5\" (UniqueName: \"kubernetes.io/projected/9760fd20-c563-4f5a-be5a-eeca1a7898a3-kube-api-access-ft9q5\") pod \"9760fd20-c563-4f5a-be5a-eeca1a7898a3\" (UID: \"9760fd20-c563-4f5a-be5a-eeca1a7898a3\") "
Oct 31 23:24:48.654557 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.654495    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9760fd20-c563-4f5a-be5a-eeca1a7898a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9760fd20-c563-4f5a-be5a-eeca1a7898a3-maas-api-openshift-ca\") pod \"9760fd20-c563-4f5a-be5a-eeca1a7898a3\" (UID: \"9760fd20-c563-4f5a-be5a-eeca1a7898a3\") "
Oct 31 23:24:48.654727 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.654632    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9760fd20-c563-4f5a-be5a-eeca1a7898a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9760fd20-c563-4f5a-be5a-eeca1a7898a3-maas-api-xks-ca\") pod \"9760fd20-c563-4f5a-be5a-eeca1a7898a3\" (UID: \"9760fd20-c563-4f5a-be5a-eeca1a7898a3\") "
Oct 31 23:24:48.655050 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.655020    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9760fd20-c563-4f5a-be5a-eeca1a7898a3-maas-api-openshift-ca" pod "9760fd20-c563-4f5a-be5a-eeca1a7898a3" (UID: "9760fd20-c563-4f5a-be5a-eeca1a7898a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:24:48.655268 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.655245    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9760fd20-c563-4f5a-be5a-eeca1a7898a3-maas-api-openshift-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:48.657096 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.657067    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9760fd20-c563-4f5a-be5a-eeca1a7898a3-kube-api-access-ft9q5" pod "9760fd20-c563-4f5a-be5a-eeca1a7898a3" (UID: "9760fd20-c563-4f5a-be5a-eeca1a7898a3"). InnerVolumeSpecName "kube-api-access-ft9q5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:48.657203 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.657133    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9760fd20-c563-4f5a-be5a-eeca1a7898a3-maas-api-xks-ca" pod "9760fd20-c563-4f5a-be5a-eeca1a7898a3" (UID: "9760fd20-c563-4f5a-be5a-eeca1a7898a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:24:48.757039 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.757006    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft9q5\" (UniqueName: \"kubernetes.io/projected/9760fd20-c563-4f5a-be5a-eeca1a7898a3-kube-api-access-ft9q5\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:48.757039 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.757041    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9760fd20-c563-4f5a-be5a-eeca1a7898a3-maas-api-xks-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:48.809772 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.809743    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:24:48.852188 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.852155    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:24:48.914291 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.914263    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:24:48.975383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.963324    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/394e1575-7171-45a2-8d19-ef551eaad0ea-istio-token\" (UniqueName: \"kubernetes.io/projected/394e1575-7171-45a2-8d19-ef551eaad0ea-istio-token\") pod \"394e1575-7171-45a2-8d19-ef551eaad0ea\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") "
Oct 31 23:24:48.975383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.963403    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-credential-socket\") pod \"bba885b3-4cbf-4677-bef9-94f492ca8794\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") "
Oct 31 23:24:48.975383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.963437    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-workload-certs\") pod \"bba885b3-4cbf-4677-bef9-94f492ca8794\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") "
Oct 31 23:24:48.975383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.963500    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-istio-envoy\") pod \"394e1575-7171-45a2-8d19-ef551eaad0ea\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") "
Oct 31 23:24:48.975383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.963544    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bba885b3-4cbf-4677-bef9-94f492ca8794-istio-token\" (UniqueName: \"kubernetes.io/projected/bba885b3-4cbf-4677-bef9-94f492ca8794-istio-token\") pod \"bba885b3-4cbf-4677-bef9-94f492ca8794\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") "
Oct 31 23:24:48.975383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.963643    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-istio-data\") pod \"bba885b3-4cbf-4677-bef9-94f492ca8794\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") "
Oct 31 23:24:48.975383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.963684    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-credential-socket\") pod \"394e1575-7171-45a2-8d19-ef551eaad0ea\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") "
Oct 31 23:24:48.975383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.963724    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bba885b3-4cbf-4677-bef9-94f492ca8794-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba885b3-4cbf-4677-bef9-94f492ca8794-istio-podinfo\") pod \"bba885b3-4cbf-4677-bef9-94f492ca8794\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") "
Oct 31 23:24:48.975383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.963764    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/394e1575-7171-45a2-8d19-ef551eaad0ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/394e1575-7171-45a2-8d19-ef551eaad0ea-istio-podinfo\") pod \"394e1575-7171-45a2-8d19-ef551eaad0ea\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") "
Oct 31 23:24:48.975383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.963830    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bba885b3-4cbf-4677-bef9-94f492ca8794-kube-api-access-m8jrv\" (UniqueName: \"kubernetes.io/projected/bba885b3-4cbf-4677-bef9-94f492ca8794-kube-api-access-m8jrv\") pod \"bba885b3-4cbf-4677-bef9-94f492ca8794\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") "
Oct 31 23:24:48.975383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.963883    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/394e1575-7171-45a2-8d19-ef551eaad0ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/394e1575-7171-45a2-8d19-ef551eaad0ea-istiod-ca-cert\") pod \"394e1575-7171-45a2-8d19-ef551eaad0ea\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") "
Oct 31 23:24:48.975383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.963922    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-workload-certs\") pod \"394e1575-7171-45a2-8d19-ef551eaad0ea\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") "
Oct 31 23:24:48.975383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.963978    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/394e1575-7171-45a2-8d19-ef551eaad0ea-kube-api-access-qnb5n\" (UniqueName: \"kubernetes.io/projected/394e1575-7171-45a2-8d19-ef551eaad0ea-kube-api-access-qnb5n\") pod \"394e1575-7171-45a2-8d19-ef551eaad0ea\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") "
Oct 31 23:24:48.975383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.964048    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-istio-data\") pod \"394e1575-7171-45a2-8d19-ef551eaad0ea\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") "
Oct 31 23:24:48.975383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.964108    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-workload-socket\") pod \"bba885b3-4cbf-4677-bef9-94f492ca8794\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") "
Oct 31 23:24:48.975383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.964174    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-istio-envoy\") pod \"bba885b3-4cbf-4677-bef9-94f492ca8794\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") "
Oct 31 23:24:48.975383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.964238    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bba885b3-4cbf-4677-bef9-94f492ca8794-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba885b3-4cbf-4677-bef9-94f492ca8794-istiod-ca-cert\") pod \"bba885b3-4cbf-4677-bef9-94f492ca8794\" (UID: \"bba885b3-4cbf-4677-bef9-94f492ca8794\") "
Oct 31 23:24:48.975383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.964281    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-workload-socket\") pod \"394e1575-7171-45a2-8d19-ef551eaad0ea\" (UID: \"394e1575-7171-45a2-8d19-ef551eaad0ea\") "
Oct 31 23:24:48.992932 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.989730    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-istio-data" pod "394e1575-7171-45a2-8d19-ef551eaad0ea" (UID: "394e1575-7171-45a2-8d19-ef551eaad0ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:48.992932 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.991337    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394e1575-7171-45a2-8d19-ef551eaad0ea-istiod-ca-cert" pod "394e1575-7171-45a2-8d19-ef551eaad0ea" (UID: "394e1575-7171-45a2-8d19-ef551eaad0ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:24:48.992932 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.991755    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-workload-certs" pod "394e1575-7171-45a2-8d19-ef551eaad0ea" (UID: "394e1575-7171-45a2-8d19-ef551eaad0ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:49.000704 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:48.998313    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-workload-socket" pod "bba885b3-4cbf-4677-bef9-94f492ca8794" (UID: "bba885b3-4cbf-4677-bef9-94f492ca8794"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:49.004623 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.003441    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bba885b3-4cbf-4677-bef9-94f492ca8794-istiod-ca-cert" pod "bba885b3-4cbf-4677-bef9-94f492ca8794" (UID: "bba885b3-4cbf-4677-bef9-94f492ca8794"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:24:49.004623 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.003739    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-credential-socket" pod "394e1575-7171-45a2-8d19-ef551eaad0ea" (UID: "394e1575-7171-45a2-8d19-ef551eaad0ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:49.004623 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.003924    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-istio-data" pod "bba885b3-4cbf-4677-bef9-94f492ca8794" (UID: "bba885b3-4cbf-4677-bef9-94f492ca8794"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:49.005885 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.005226    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-workload-certs" pod "bba885b3-4cbf-4677-bef9-94f492ca8794" (UID: "bba885b3-4cbf-4677-bef9-94f492ca8794"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:49.005885 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.005438    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-credential-socket" pod "bba885b3-4cbf-4677-bef9-94f492ca8794" (UID: "bba885b3-4cbf-4677-bef9-94f492ca8794"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:49.008506 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.008482    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-workload-socket" pod "394e1575-7171-45a2-8d19-ef551eaad0ea" (UID: "394e1575-7171-45a2-8d19-ef551eaad0ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:49.010904 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.010874    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/394e1575-7171-45a2-8d19-ef551eaad0ea-istio-podinfo" pod "394e1575-7171-45a2-8d19-ef551eaad0ea" (UID: "394e1575-7171-45a2-8d19-ef551eaad0ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:24:49.011010 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.010896    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/394e1575-7171-45a2-8d19-ef551eaad0ea-istio-token" pod "394e1575-7171-45a2-8d19-ef551eaad0ea" (UID: "394e1575-7171-45a2-8d19-ef551eaad0ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:49.011110 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.011090    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"]
Oct 31 23:24:49.011282 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.011249    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bba885b3-4cbf-4677-bef9-94f492ca8794-istio-podinfo" pod "bba885b3-4cbf-4677-bef9-94f492ca8794" (UID: "bba885b3-4cbf-4677-bef9-94f492ca8794"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:24:49.011531 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.011508    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-istio-envoy" pod "394e1575-7171-45a2-8d19-ef551eaad0ea" (UID: "394e1575-7171-45a2-8d19-ef551eaad0ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:49.023410 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.021427    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"]
Oct 31 23:24:49.031258 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.030858    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba885b3-4cbf-4677-bef9-94f492ca8794-istio-token" pod "bba885b3-4cbf-4677-bef9-94f492ca8794" (UID: "bba885b3-4cbf-4677-bef9-94f492ca8794"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:49.034899 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.034852    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba885b3-4cbf-4677-bef9-94f492ca8794-kube-api-access-m8jrv" pod "bba885b3-4cbf-4677-bef9-94f492ca8794" (UID: "bba885b3-4cbf-4677-bef9-94f492ca8794"). InnerVolumeSpecName "kube-api-access-m8jrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:49.035164 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.035126    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/394e1575-7171-45a2-8d19-ef551eaad0ea-kube-api-access-qnb5n" pod "394e1575-7171-45a2-8d19-ef551eaad0ea" (UID: "394e1575-7171-45a2-8d19-ef551eaad0ea"). InnerVolumeSpecName "kube-api-access-qnb5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:49.048978 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.048907    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-istio-envoy" pod "bba885b3-4cbf-4677-bef9-94f492ca8794" (UID: "bba885b3-4cbf-4677-bef9-94f492ca8794"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:49.066703 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.065464    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-kube-api-access-t9tbw\" (UniqueName: \"kubernetes.io/projected/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-kube-api-access-t9tbw\") pod \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") "
Oct 31 23:24:49.066703 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.065525    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istio-data\") pod \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") "
Oct 31 23:24:49.066703 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.065658    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istio-envoy\") pod \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") "
Oct 31 23:24:49.066703 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.065720    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-workload-certs\") pod \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") "
Oct 31 23:24:49.066703 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.065773    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istio-token\" (UniqueName: \"kubernetes.io/projected/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istio-token\") pod \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") "
Oct 31 23:24:49.066703 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.065817    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-credential-socket\") pod \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") "
Oct 31 23:24:49.066703 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.065885    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istiod-ca-cert\") pod \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") "
Oct 31 23:24:49.066703 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.065946    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-workload-socket\") pod \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") "
Oct 31 23:24:49.066703 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.065980    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istio-podinfo\") pod \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\" (UID: \"2b5f9bbb-897d-44dc-9eb8-5887091b4d64\") "
Oct 31 23:24:49.066703 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.066458    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-workload-certs" pod "2b5f9bbb-897d-44dc-9eb8-5887091b4d64" (UID: "2b5f9bbb-897d-44dc-9eb8-5887091b4d64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:49.066703 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.066677    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-credential-socket" pod "2b5f9bbb-897d-44dc-9eb8-5887091b4d64" (UID: "2b5f9bbb-897d-44dc-9eb8-5887091b4d64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:49.067351 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.066712    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:49.067351 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.066751    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:49.067351 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.066767    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:49.067351 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.066782    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba885b3-4cbf-4677-bef9-94f492ca8794-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:49.067351 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.066793    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istiod-ca-cert" pod "2b5f9bbb-897d-44dc-9eb8-5887091b4d64" (UID: "2b5f9bbb-897d-44dc-9eb8-5887091b4d64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:24:49.067351 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.066798    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:49.067351 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.066831    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/394e1575-7171-45a2-8d19-ef551eaad0ea-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:49.067351 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.066842    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:49.067351 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.066851    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:49.067351 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.066860    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:49.067351 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.066869    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba885b3-4cbf-4677-bef9-94f492ca8794-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:49.067351 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.066877    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba885b3-4cbf-4677-bef9-94f492ca8794-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:49.067351 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.066887    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:49.067351 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.066895    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba885b3-4cbf-4677-bef9-94f492ca8794-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:49.067351 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.066904    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/394e1575-7171-45a2-8d19-ef551eaad0ea-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:49.067351 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.066913    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8jrv\" (UniqueName: \"kubernetes.io/projected/bba885b3-4cbf-4677-bef9-94f492ca8794-kube-api-access-m8jrv\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:49.067351 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.066921    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/394e1575-7171-45a2-8d19-ef551eaad0ea-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:49.067351 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.066930    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/394e1575-7171-45a2-8d19-ef551eaad0ea-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:49.067351 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.066939    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnb5n\" (UniqueName: \"kubernetes.io/projected/394e1575-7171-45a2-8d19-ef551eaad0ea-kube-api-access-qnb5n\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:49.067351 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.066948    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:49.067351 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.067134    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-workload-socket" pod "2b5f9bbb-897d-44dc-9eb8-5887091b4d64" (UID: "2b5f9bbb-897d-44dc-9eb8-5887091b4d64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:49.067351 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.067310    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istio-data" pod "2b5f9bbb-897d-44dc-9eb8-5887091b4d64" (UID: "2b5f9bbb-897d-44dc-9eb8-5887091b4d64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:49.069969 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.069730    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istio-token" pod "2b5f9bbb-897d-44dc-9eb8-5887091b4d64" (UID: "2b5f9bbb-897d-44dc-9eb8-5887091b4d64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:49.070935 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.070907    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istio-podinfo" pod "2b5f9bbb-897d-44dc-9eb8-5887091b4d64" (UID: "2b5f9bbb-897d-44dc-9eb8-5887091b4d64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:24:49.073570 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.073534    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istio-envoy" pod "2b5f9bbb-897d-44dc-9eb8-5887091b4d64" (UID: "2b5f9bbb-897d-44dc-9eb8-5887091b4d64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:49.073743 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.073723    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-kube-api-access-t9tbw" pod "2b5f9bbb-897d-44dc-9eb8-5887091b4d64" (UID: "2b5f9bbb-897d-44dc-9eb8-5887091b4d64"). InnerVolumeSpecName "kube-api-access-t9tbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:49.168848 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.168762    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9tbw\" (UniqueName: \"kubernetes.io/projected/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-kube-api-access-t9tbw\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:49.168848 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.168802    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:49.168848 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.168818    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:49.168848 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.168832    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:49.168848 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.168845    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:49.169128 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.168859    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:49.169128 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.168871    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:49.169128 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.168885    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5f9bbb-897d-44dc-9eb8-5887091b4d64-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:49.359661 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.359615    4301 generic.go:356] "Generic (PLEG): container finished" podID="2b5f9bbb-897d-44dc-9eb8-5887091b4d64" containerID="c46aff3eccf6248c32249539e8a9da8fd01a605fcafdda31143efae0804002ea" exitCode=0
Oct 31 23:24:49.359844 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.359678    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"
Oct 31 23:24:49.359844 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.359697    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf" event={"ID":"2b5f9bbb-897d-44dc-9eb8-5887091b4d64","Type":"ContainerDied","Data":"c46aff3eccf6248c32249539e8a9da8fd01a605fcafdda31143efae0804002ea"}
Oct 31 23:24:49.359844 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.359789    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf" event={"ID":"2b5f9bbb-897d-44dc-9eb8-5887091b4d64","Type":"ContainerDied","Data":"5b10574ca343e10693f9ff4e61da84299bb5bc2de587da3c05fb850c250419d6"}
Oct 31 23:24:49.359844 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.359821    4301 scope.go:122] "RemoveContainer" containerID="c46aff3eccf6248c32249539e8a9da8fd01a605fcafdda31143efae0804002ea"
Oct 31 23:24:49.361325 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.361244    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6d2120-x6jhx"
Oct 31 23:24:49.361325 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.361267    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6d2120-x6jhx" event={"ID":"9760fd20-c563-4f5a-be5a-eeca1a7898a3","Type":"ContainerDied","Data":"113b2ab6036a280661d382d8d6150bafd74cfcb212a128dd01ec68ea42bdf3fa"}
Oct 31 23:24:49.361325 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.361307    4301 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="113b2ab6036a280661d382d8d6150bafd74cfcb212a128dd01ec68ea42bdf3fa"
Oct 31 23:24:49.363258 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.363222    4301 generic.go:356] "Generic (PLEG): container finished" podID="394e1575-7171-45a2-8d19-ef551eaad0ea" containerID="b17227da4c1379614a3ac17d6f7bc0e0c17b0b4b9f4fae005652472059b2797f" exitCode=0
Oct 31 23:24:49.363389 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.363253    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p" event={"ID":"394e1575-7171-45a2-8d19-ef551eaad0ea","Type":"ContainerDied","Data":"b17227da4c1379614a3ac17d6f7bc0e0c17b0b4b9f4fae005652472059b2797f"}
Oct 31 23:24:49.363389 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.363306    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p" event={"ID":"394e1575-7171-45a2-8d19-ef551eaad0ea","Type":"ContainerDied","Data":"d7932243ce9c51138cde57a996faf8fc01aa06e71558fcd729f1ec4bdfba07d4"}
Oct 31 23:24:49.363389 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.363269    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"
Oct 31 23:24:49.364895 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.364871    4301 generic.go:356] "Generic (PLEG): container finished" podID="bba885b3-4cbf-4677-bef9-94f492ca8794" containerID="004c101df93af2590ce5f0f449ca6129709bb62dbf8db2245e80369b76c4b0e0" exitCode=0
Oct 31 23:24:49.365022 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.364905    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2" event={"ID":"bba885b3-4cbf-4677-bef9-94f492ca8794","Type":"ContainerDied","Data":"004c101df93af2590ce5f0f449ca6129709bb62dbf8db2245e80369b76c4b0e0"}
Oct 31 23:24:49.365022 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.364916    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"
Oct 31 23:24:49.365022 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.364933    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2" event={"ID":"bba885b3-4cbf-4677-bef9-94f492ca8794","Type":"ContainerDied","Data":"1177c510debc0a4a27c849e2d60362d7cdd27f974e52dab9b5553c86688ef986"}
Oct 31 23:24:49.374126 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.374109    4301 scope.go:122] "RemoveContainer" containerID="c46aff3eccf6248c32249539e8a9da8fd01a605fcafdda31143efae0804002ea"
Oct 31 23:24:49.375230 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:24:49.375206    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c46aff3eccf6248c32249539e8a9da8fd01a605fcafdda31143efae0804002ea\": container with ID starting with c46aff3eccf6248c32249539e8a9da8fd01a605fcafdda31143efae0804002ea not found: ID does not exist" containerID="c46aff3eccf6248c32249539e8a9da8fd01a605fcafdda31143efae0804002ea"
Oct 31 23:24:49.375464 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.375240    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c46aff3eccf6248c32249539e8a9da8fd01a605fcafdda31143efae0804002ea"} err="failed to get container status \"c46aff3eccf6248c32249539e8a9da8fd01a605fcafdda31143efae0804002ea\": rpc error: code = NotFound desc = could not find container \"c46aff3eccf6248c32249539e8a9da8fd01a605fcafdda31143efae0804002ea\": container with ID starting with c46aff3eccf6248c32249539e8a9da8fd01a605fcafdda31143efae0804002ea not found: ID does not exist"
Oct 31 23:24:49.375464 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.375265    4301 scope.go:122] "RemoveContainer" containerID="b17227da4c1379614a3ac17d6f7bc0e0c17b0b4b9f4fae005652472059b2797f"
Oct 31 23:24:49.388190 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.388171    4301 scope.go:122] "RemoveContainer" containerID="0f85ac85e722e495f38549d8df0595653b60c874b0a2047a4ac9e9aed8bf6864"
Oct 31 23:24:49.392843 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.392819    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"]
Oct 31 23:24:49.399895 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.399850    4301 scope.go:122] "RemoveContainer" containerID="b17227da4c1379614a3ac17d6f7bc0e0c17b0b4b9f4fae005652472059b2797f"
Oct 31 23:24:49.400165 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:24:49.400143    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b17227da4c1379614a3ac17d6f7bc0e0c17b0b4b9f4fae005652472059b2797f\": container with ID starting with b17227da4c1379614a3ac17d6f7bc0e0c17b0b4b9f4fae005652472059b2797f not found: ID does not exist" containerID="b17227da4c1379614a3ac17d6f7bc0e0c17b0b4b9f4fae005652472059b2797f"
Oct 31 23:24:49.400239 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.400174    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b17227da4c1379614a3ac17d6f7bc0e0c17b0b4b9f4fae005652472059b2797f"} err="failed to get container status \"b17227da4c1379614a3ac17d6f7bc0e0c17b0b4b9f4fae005652472059b2797f\": rpc error: code = NotFound desc = could not find container \"b17227da4c1379614a3ac17d6f7bc0e0c17b0b4b9f4fae005652472059b2797f\": container with ID starting with b17227da4c1379614a3ac17d6f7bc0e0c17b0b4b9f4fae005652472059b2797f not found: ID does not exist"
Oct 31 23:24:49.400239 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.400190    4301 scope.go:122] "RemoveContainer" containerID="0f85ac85e722e495f38549d8df0595653b60c874b0a2047a4ac9e9aed8bf6864"
Oct 31 23:24:49.400528 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:24:49.400510    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f85ac85e722e495f38549d8df0595653b60c874b0a2047a4ac9e9aed8bf6864\": container with ID starting with 0f85ac85e722e495f38549d8df0595653b60c874b0a2047a4ac9e9aed8bf6864 not found: ID does not exist" containerID="0f85ac85e722e495f38549d8df0595653b60c874b0a2047a4ac9e9aed8bf6864"
Oct 31 23:24:49.400581 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.400532    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f85ac85e722e495f38549d8df0595653b60c874b0a2047a4ac9e9aed8bf6864"} err="failed to get container status \"0f85ac85e722e495f38549d8df0595653b60c874b0a2047a4ac9e9aed8bf6864\": rpc error: code = NotFound desc = could not find container \"0f85ac85e722e495f38549d8df0595653b60c874b0a2047a4ac9e9aed8bf6864\": container with ID starting with 0f85ac85e722e495f38549d8df0595653b60c874b0a2047a4ac9e9aed8bf6864 not found: ID does not exist"
Oct 31 23:24:49.400581 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.400545    4301 scope.go:122] "RemoveContainer" containerID="004c101df93af2590ce5f0f449ca6129709bb62dbf8db2245e80369b76c4b0e0"
Oct 31 23:24:49.406195 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.406155    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-ns5bf"]
Oct 31 23:24:49.411158 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.411137    4301 scope.go:122] "RemoveContainer" containerID="004c101df93af2590ce5f0f449ca6129709bb62dbf8db2245e80369b76c4b0e0"
Oct 31 23:24:49.411554 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:24:49.411532    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"004c101df93af2590ce5f0f449ca6129709bb62dbf8db2245e80369b76c4b0e0\": container with ID starting with 004c101df93af2590ce5f0f449ca6129709bb62dbf8db2245e80369b76c4b0e0 not found: ID does not exist" containerID="004c101df93af2590ce5f0f449ca6129709bb62dbf8db2245e80369b76c4b0e0"
Oct 31 23:24:49.411638 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.411563    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"004c101df93af2590ce5f0f449ca6129709bb62dbf8db2245e80369b76c4b0e0"} err="failed to get container status \"004c101df93af2590ce5f0f449ca6129709bb62dbf8db2245e80369b76c4b0e0\": rpc error: code = NotFound desc = could not find container \"004c101df93af2590ce5f0f449ca6129709bb62dbf8db2245e80369b76c4b0e0\": container with ID starting with 004c101df93af2590ce5f0f449ca6129709bb62dbf8db2245e80369b76c4b0e0 not found: ID does not exist"
Oct 31 23:24:49.411936 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.411923    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"]
Oct 31 23:24:49.414977 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.414957    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vb48p"]
Oct 31 23:24:49.425491 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.425424    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"]
Oct 31 23:24:49.429590 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.429573    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-9lvq2"]
Oct 31 23:24:49.608891 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.608856    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main" containerID="cri-o://5890f9d31e51eda37394c7bcc3dc59d847ab904af5ae24c0c9a4bf14aa6a5cc2" gracePeriod=45
Oct 31 23:24:49.951631 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:49.951598    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:24:50.094315 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.094225    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2a4d961-adc7-4f38-9825-33f1230ddadb-home\" (UniqueName: \"kubernetes.io/empty-dir/f2a4d961-adc7-4f38-9825-33f1230ddadb-home\") pod \"f2a4d961-adc7-4f38-9825-33f1230ddadb\" (UID: \"f2a4d961-adc7-4f38-9825-33f1230ddadb\") "
Oct 31 23:24:50.094315 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.094298    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2a4d961-adc7-4f38-9825-33f1230ddadb-kube-api-access-dlxsj\" (UniqueName: \"kubernetes.io/projected/f2a4d961-adc7-4f38-9825-33f1230ddadb-kube-api-access-dlxsj\") pod \"f2a4d961-adc7-4f38-9825-33f1230ddadb\" (UID: \"f2a4d961-adc7-4f38-9825-33f1230ddadb\") "
Oct 31 23:24:50.094598 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.094350    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2a4d961-adc7-4f38-9825-33f1230ddadb-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2a4d961-adc7-4f38-9825-33f1230ddadb-tls-certs\") pod \"f2a4d961-adc7-4f38-9825-33f1230ddadb\" (UID: \"f2a4d961-adc7-4f38-9825-33f1230ddadb\") "
Oct 31 23:24:50.094598 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.094443    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2a4d961-adc7-4f38-9825-33f1230ddadb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2a4d961-adc7-4f38-9825-33f1230ddadb-model-cache\") pod \"f2a4d961-adc7-4f38-9825-33f1230ddadb\" (UID: \"f2a4d961-adc7-4f38-9825-33f1230ddadb\") "
Oct 31 23:24:50.094598 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.094495    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2a4d961-adc7-4f38-9825-33f1230ddadb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2a4d961-adc7-4f38-9825-33f1230ddadb-tmp-dir\") pod \"f2a4d961-adc7-4f38-9825-33f1230ddadb\" (UID: \"f2a4d961-adc7-4f38-9825-33f1230ddadb\") "
Oct 31 23:24:50.094598 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.094597    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2a4d961-adc7-4f38-9825-33f1230ddadb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2a4d961-adc7-4f38-9825-33f1230ddadb-dshm\") pod \"f2a4d961-adc7-4f38-9825-33f1230ddadb\" (UID: \"f2a4d961-adc7-4f38-9825-33f1230ddadb\") "
Oct 31 23:24:50.094825 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.094645    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a4d961-adc7-4f38-9825-33f1230ddadb-home" pod "f2a4d961-adc7-4f38-9825-33f1230ddadb" (UID: "f2a4d961-adc7-4f38-9825-33f1230ddadb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:50.095437 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.095387    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a4d961-adc7-4f38-9825-33f1230ddadb-tmp-dir" pod "f2a4d961-adc7-4f38-9825-33f1230ddadb" (UID: "f2a4d961-adc7-4f38-9825-33f1230ddadb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:50.095571 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.095468    4301 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2a4d961-adc7-4f38-9825-33f1230ddadb-home\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:50.095657 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.095454    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a4d961-adc7-4f38-9825-33f1230ddadb-model-cache" pod "f2a4d961-adc7-4f38-9825-33f1230ddadb" (UID: "f2a4d961-adc7-4f38-9825-33f1230ddadb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:50.097951 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.097923    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a4d961-adc7-4f38-9825-33f1230ddadb-dshm" pod "f2a4d961-adc7-4f38-9825-33f1230ddadb" (UID: "f2a4d961-adc7-4f38-9825-33f1230ddadb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:50.098233 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.098213    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2a4d961-adc7-4f38-9825-33f1230ddadb-tls-certs" pod "f2a4d961-adc7-4f38-9825-33f1230ddadb" (UID: "f2a4d961-adc7-4f38-9825-33f1230ddadb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:24:50.098306 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.098233    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2a4d961-adc7-4f38-9825-33f1230ddadb-kube-api-access-dlxsj" pod "f2a4d961-adc7-4f38-9825-33f1230ddadb" (UID: "f2a4d961-adc7-4f38-9825-33f1230ddadb"). InnerVolumeSpecName "kube-api-access-dlxsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:50.196883 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.196849    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlxsj\" (UniqueName: \"kubernetes.io/projected/f2a4d961-adc7-4f38-9825-33f1230ddadb-kube-api-access-dlxsj\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:50.196883 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.196890    4301 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2a4d961-adc7-4f38-9825-33f1230ddadb-tls-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:50.197094 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.196906    4301 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2a4d961-adc7-4f38-9825-33f1230ddadb-model-cache\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:50.197094 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.196920    4301 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2a4d961-adc7-4f38-9825-33f1230ddadb-tmp-dir\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:50.197094 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.196931    4301 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2a4d961-adc7-4f38-9825-33f1230ddadb-dshm\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:50.373073 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.373036    4301 generic.go:356] "Generic (PLEG): container finished" podID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerID="5890f9d31e51eda37394c7bcc3dc59d847ab904af5ae24c0c9a4bf14aa6a5cc2" exitCode=0
Oct 31 23:24:50.373252 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.373092    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"
Oct 31 23:24:50.373252 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.373117    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" event={"ID":"f2a4d961-adc7-4f38-9825-33f1230ddadb","Type":"ContainerDied","Data":"5890f9d31e51eda37394c7bcc3dc59d847ab904af5ae24c0c9a4bf14aa6a5cc2"}
Oct 31 23:24:50.373252 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.373162    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8" event={"ID":"f2a4d961-adc7-4f38-9825-33f1230ddadb","Type":"ContainerDied","Data":"de04a4f4bd0a3fdccdf7e42c3c282b2189c5f0419d425af7871cec44a58fa2d6"}
Oct 31 23:24:50.373252 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.373186    4301 scope.go:122] "RemoveContainer" containerID="5890f9d31e51eda37394c7bcc3dc59d847ab904af5ae24c0c9a4bf14aa6a5cc2"
Oct 31 23:24:50.386729 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.386704    4301 scope.go:122] "RemoveContainer" containerID="5890f9d31e51eda37394c7bcc3dc59d847ab904af5ae24c0c9a4bf14aa6a5cc2"
Oct 31 23:24:50.387022 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:24:50.387002    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5890f9d31e51eda37394c7bcc3dc59d847ab904af5ae24c0c9a4bf14aa6a5cc2\": container with ID starting with 5890f9d31e51eda37394c7bcc3dc59d847ab904af5ae24c0c9a4bf14aa6a5cc2 not found: ID does not exist" containerID="5890f9d31e51eda37394c7bcc3dc59d847ab904af5ae24c0c9a4bf14aa6a5cc2"
Oct 31 23:24:50.387142 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.387030    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5890f9d31e51eda37394c7bcc3dc59d847ab904af5ae24c0c9a4bf14aa6a5cc2"} err="failed to get container status \"5890f9d31e51eda37394c7bcc3dc59d847ab904af5ae24c0c9a4bf14aa6a5cc2\": rpc error: code = NotFound desc = could not find container \"5890f9d31e51eda37394c7bcc3dc59d847ab904af5ae24c0c9a4bf14aa6a5cc2\": container with ID starting with 5890f9d31e51eda37394c7bcc3dc59d847ab904af5ae24c0c9a4bf14aa6a5cc2 not found: ID does not exist"
Oct 31 23:24:50.419182 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.419151    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"]
Oct 31 23:24:50.458125 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.458088    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-7d7eed/distinct-2-w6-7d7eed-kserve-7668d695cd-fd4m8"]
Oct 31 23:24:50.461602 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.461417    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" containerID="cri-o://03db228446522bf212a08d53b5d2d1d60542e2a3767b08327929de6e374b0f3d" gracePeriod=45
Oct 31 23:24:50.493378 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.493256    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" containerID="cri-o://aaeaff5fa1f733c19d17d6f7f7af6c9b749894159431c70fb2f4004c78cb9092" gracePeriod=45
Oct 31 23:24:50.495235 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.495180    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" containerID="cri-o://59f03b727cae67bed70d755db3db359e263bf6397d33b55041a696e1a0ed47b5" gracePeriod=45
Oct 31 23:24:50.498209 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.498148    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main" containerID="cri-o://cd1a83d5f8ec000fbccaaa6a674ea6b1967c3ad6015a9c1c35dbd6f9b1dc9b55" gracePeriod=45
Oct 31 23:24:50.503324 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.503202    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": read tcp 10.133.0.2:35266->10.133.0.36:8000: read: connection reset by peer"
Oct 31 23:24:50.522100 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.522066    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": dial tcp 10.133.0.61:8000: connect: connection refused"
Oct 31 23:24:50.547209 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.547174    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": dial tcp 10.133.0.56:8000: connect: connection refused"
Oct 31 23:24:50.656729 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.656698    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"]
Oct 31 23:24:50.657468 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:50.657436    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx" podUID="9a58ab83-2394-48a8-aa4f-04c22ee67a24" containerName="istio-proxy" containerID="cri-o://0b88e034f850cc37100ede8de632d0db0f238582eb0c3975b27dea1b6a615860" gracePeriod=30
Oct 31 23:24:51.089273 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.089233    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:24:51.096573 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.096552    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:24:51.098607 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.098589    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:24:51.105412 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.105394    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:24:51.123418 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.123390    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:24:51.247853 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.247764    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa98bd9c-f413-4852-869c-adc53259a977-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd9c-f413-4852-869c-adc53259a977-model-cache\") pod \"fa98bd9c-f413-4852-869c-adc53259a977\" (UID: \"fa98bd9c-f413-4852-869c-adc53259a977\") "
Oct 31 23:24:51.247853 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.247810    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-workload-socket\") pod \"e9de1723-3c15-4c53-8080-26e27ac118e4\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") "
Oct 31 23:24:51.248074 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.247873    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa98bd9c-f413-4852-869c-adc53259a977-tls-certs\" (UniqueName: \"kubernetes.io/secret/fa98bd9c-f413-4852-869c-adc53259a977-tls-certs\") pod \"fa98bd9c-f413-4852-869c-adc53259a977\" (UID: \"fa98bd9c-f413-4852-869c-adc53259a977\") "
Oct 31 23:24:51.248074 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.247899    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-dshm\") pod \"289e47fa-d3c8-49b4-aac7-0cd0959e64aa\" (UID: \"289e47fa-d3c8-49b4-aac7-0cd0959e64aa\") "
Oct 31 23:24:51.248074 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.247937    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9de1723-3c15-4c53-8080-26e27ac118e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9de1723-3c15-4c53-8080-26e27ac118e4-istiod-ca-cert\") pod \"e9de1723-3c15-4c53-8080-26e27ac118e4\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") "
Oct 31 23:24:51.248074 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.247973    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa98bd9c-f413-4852-869c-adc53259a977-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd9c-f413-4852-869c-adc53259a977-tmp-dir\") pod \"fa98bd9c-f413-4852-869c-adc53259a977\" (UID: \"fa98bd9c-f413-4852-869c-adc53259a977\") "
Oct 31 23:24:51.248074 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.248001    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9de1723-3c15-4c53-8080-26e27ac118e4-kube-api-access-b8844\" (UniqueName: \"kubernetes.io/projected/e9de1723-3c15-4c53-8080-26e27ac118e4-kube-api-access-b8844\") pod \"e9de1723-3c15-4c53-8080-26e27ac118e4\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") "
Oct 31 23:24:51.248074 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.248038    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca870469-c464-4272-a7d7-61081a3d2afc-kube-api-access-8vx7b\" (UniqueName: \"kubernetes.io/projected/ca870469-c464-4272-a7d7-61081a3d2afc-kube-api-access-8vx7b\") pod \"ca870469-c464-4272-a7d7-61081a3d2afc\" (UID: \"ca870469-c464-4272-a7d7-61081a3d2afc\") "
Oct 31 23:24:51.248074 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.248070    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-tmp-dir\") pod \"289e47fa-d3c8-49b4-aac7-0cd0959e64aa\" (UID: \"289e47fa-d3c8-49b4-aac7-0cd0959e64aa\") "
Oct 31 23:24:51.248503 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.248096    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca870469-c464-4272-a7d7-61081a3d2afc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca870469-c464-4272-a7d7-61081a3d2afc-tmp-dir\") pod \"ca870469-c464-4272-a7d7-61081a3d2afc\" (UID: \"ca870469-c464-4272-a7d7-61081a3d2afc\") "
Oct 31 23:24:51.248503 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.248096    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa98bd9c-f413-4852-869c-adc53259a977-model-cache" pod "fa98bd9c-f413-4852-869c-adc53259a977" (UID: "fa98bd9c-f413-4852-869c-adc53259a977"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:51.248503 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.248137    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-istio-envoy\") pod \"e9de1723-3c15-4c53-8080-26e27ac118e4\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") "
Oct 31 23:24:51.248503 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.248159    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-workload-socket" pod "e9de1723-3c15-4c53-8080-26e27ac118e4" (UID: "e9de1723-3c15-4c53-8080-26e27ac118e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:51.248503 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.248165    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/595ded14-5c69-427e-a232-e969122cf00f-tls-certs\" (UniqueName: \"kubernetes.io/secret/595ded14-5c69-427e-a232-e969122cf00f-tls-certs\") pod \"595ded14-5c69-427e-a232-e969122cf00f\" (UID: \"595ded14-5c69-427e-a232-e969122cf00f\") "
Oct 31 23:24:51.248503 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.248232    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-kube-api-access-mx8t5\" (UniqueName: \"kubernetes.io/projected/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-kube-api-access-mx8t5\") pod \"289e47fa-d3c8-49b4-aac7-0cd0959e64aa\" (UID: \"289e47fa-d3c8-49b4-aac7-0cd0959e64aa\") "
Oct 31 23:24:51.248503 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.248269    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa98bd9c-f413-4852-869c-adc53259a977-home\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd9c-f413-4852-869c-adc53259a977-home\") pod \"fa98bd9c-f413-4852-869c-adc53259a977\" (UID: \"fa98bd9c-f413-4852-869c-adc53259a977\") "
Oct 31 23:24:51.248503 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.248372    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/595ded14-5c69-427e-a232-e969122cf00f-home\" (UniqueName: \"kubernetes.io/empty-dir/595ded14-5c69-427e-a232-e969122cf00f-home\") pod \"595ded14-5c69-427e-a232-e969122cf00f\" (UID: \"595ded14-5c69-427e-a232-e969122cf00f\") "
Oct 31 23:24:51.248503 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.248421    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9de1723-3c15-4c53-8080-26e27ac118e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9de1723-3c15-4c53-8080-26e27ac118e4-istio-podinfo\") pod \"e9de1723-3c15-4c53-8080-26e27ac118e4\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") "
Oct 31 23:24:51.248503 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.248505    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-istio-data\") pod \"e9de1723-3c15-4c53-8080-26e27ac118e4\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") "
Oct 31 23:24:51.249056 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.248560    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/595ded14-5c69-427e-a232-e969122cf00f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/595ded14-5c69-427e-a232-e969122cf00f-dshm\") pod \"595ded14-5c69-427e-a232-e969122cf00f\" (UID: \"595ded14-5c69-427e-a232-e969122cf00f\") "
Oct 31 23:24:51.249056 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.248603    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/595ded14-5c69-427e-a232-e969122cf00f-kube-api-access-l74mg\" (UniqueName: \"kubernetes.io/projected/595ded14-5c69-427e-a232-e969122cf00f-kube-api-access-l74mg\") pod \"595ded14-5c69-427e-a232-e969122cf00f\" (UID: \"595ded14-5c69-427e-a232-e969122cf00f\") "
Oct 31 23:24:51.249056 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.248683    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-home\" (UniqueName: \"kubernetes.io/empty-dir/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-home\") pod \"289e47fa-d3c8-49b4-aac7-0cd0959e64aa\" (UID: \"289e47fa-d3c8-49b4-aac7-0cd0959e64aa\") "
Oct 31 23:24:51.249056 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.248710    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9de1723-3c15-4c53-8080-26e27ac118e4-istio-token\" (UniqueName: \"kubernetes.io/projected/e9de1723-3c15-4c53-8080-26e27ac118e4-istio-token\") pod \"e9de1723-3c15-4c53-8080-26e27ac118e4\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") "
Oct 31 23:24:51.249056 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.248761    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-workload-certs\") pod \"e9de1723-3c15-4c53-8080-26e27ac118e4\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") "
Oct 31 23:24:51.249056 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.248813    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca870469-c464-4272-a7d7-61081a3d2afc-home\" (UniqueName: \"kubernetes.io/empty-dir/ca870469-c464-4272-a7d7-61081a3d2afc-home\") pod \"ca870469-c464-4272-a7d7-61081a3d2afc\" (UID: \"ca870469-c464-4272-a7d7-61081a3d2afc\") "
Oct 31 23:24:51.249056 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.248842    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595ded14-5c69-427e-a232-e969122cf00f-home" pod "595ded14-5c69-427e-a232-e969122cf00f" (UID: "595ded14-5c69-427e-a232-e969122cf00f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:51.249056 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.248845    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/595ded14-5c69-427e-a232-e969122cf00f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/595ded14-5c69-427e-a232-e969122cf00f-model-cache\") pod \"595ded14-5c69-427e-a232-e969122cf00f\" (UID: \"595ded14-5c69-427e-a232-e969122cf00f\") "
Oct 31 23:24:51.249056 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.248913    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-tls-certs\" (UniqueName: \"kubernetes.io/secret/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-tls-certs\") pod \"289e47fa-d3c8-49b4-aac7-0cd0959e64aa\" (UID: \"289e47fa-d3c8-49b4-aac7-0cd0959e64aa\") "
Oct 31 23:24:51.249056 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.248963    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca870469-c464-4272-a7d7-61081a3d2afc-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca870469-c464-4272-a7d7-61081a3d2afc-tls-certs\") pod \"ca870469-c464-4272-a7d7-61081a3d2afc\" (UID: \"ca870469-c464-4272-a7d7-61081a3d2afc\") "
Oct 31 23:24:51.249056 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.249051    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595ded14-5c69-427e-a232-e969122cf00f-model-cache" pod "595ded14-5c69-427e-a232-e969122cf00f" (UID: "595ded14-5c69-427e-a232-e969122cf00f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:51.249667 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.249134    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9de1723-3c15-4c53-8080-26e27ac118e4-istiod-ca-cert" pod "e9de1723-3c15-4c53-8080-26e27ac118e4" (UID: "e9de1723-3c15-4c53-8080-26e27ac118e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:24:51.249667 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.249265    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa98bd9c-f413-4852-869c-adc53259a977-tmp-dir" pod "fa98bd9c-f413-4852-869c-adc53259a977" (UID: "fa98bd9c-f413-4852-869c-adc53259a977"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:51.249667 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.249278    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca870469-c464-4272-a7d7-61081a3d2afc-tmp-dir" pod "ca870469-c464-4272-a7d7-61081a3d2afc" (UID: "ca870469-c464-4272-a7d7-61081a3d2afc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:51.249667 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.249546    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa98bd9c-f413-4852-869c-adc53259a977-kube-api-access-8rxtm\" (UniqueName: \"kubernetes.io/projected/fa98bd9c-f413-4852-869c-adc53259a977-kube-api-access-8rxtm\") pod \"fa98bd9c-f413-4852-869c-adc53259a977\" (UID: \"fa98bd9c-f413-4852-869c-adc53259a977\") "
Oct 31 23:24:51.249667 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.249611    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/595ded14-5c69-427e-a232-e969122cf00f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/595ded14-5c69-427e-a232-e969122cf00f-tmp-dir\") pod \"595ded14-5c69-427e-a232-e969122cf00f\" (UID: \"595ded14-5c69-427e-a232-e969122cf00f\") "
Oct 31 23:24:51.249667 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.249655    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca870469-c464-4272-a7d7-61081a3d2afc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca870469-c464-4272-a7d7-61081a3d2afc-dshm\") pod \"ca870469-c464-4272-a7d7-61081a3d2afc\" (UID: \"ca870469-c464-4272-a7d7-61081a3d2afc\") "
Oct 31 23:24:51.249960 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.249691    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa98bd9c-f413-4852-869c-adc53259a977-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd9c-f413-4852-869c-adc53259a977-dshm\") pod \"fa98bd9c-f413-4852-869c-adc53259a977\" (UID: \"fa98bd9c-f413-4852-869c-adc53259a977\") "
Oct 31 23:24:51.249960 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.249717    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-credential-socket\") pod \"e9de1723-3c15-4c53-8080-26e27ac118e4\" (UID: \"e9de1723-3c15-4c53-8080-26e27ac118e4\") "
Oct 31 23:24:51.249960 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.249769    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca870469-c464-4272-a7d7-61081a3d2afc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca870469-c464-4272-a7d7-61081a3d2afc-model-cache\") pod \"ca870469-c464-4272-a7d7-61081a3d2afc\" (UID: \"ca870469-c464-4272-a7d7-61081a3d2afc\") "
Oct 31 23:24:51.249960 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.249812    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-model-cache\") pod \"289e47fa-d3c8-49b4-aac7-0cd0959e64aa\" (UID: \"289e47fa-d3c8-49b4-aac7-0cd0959e64aa\") "
Oct 31 23:24:51.250329 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.250166    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-home" pod "289e47fa-d3c8-49b4-aac7-0cd0959e64aa" (UID: "289e47fa-d3c8-49b4-aac7-0cd0959e64aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:51.250482 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.250385    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-model-cache" pod "289e47fa-d3c8-49b4-aac7-0cd0959e64aa" (UID: "289e47fa-d3c8-49b4-aac7-0cd0959e64aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:51.251062 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.250549    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-istio-data" pod "e9de1723-3c15-4c53-8080-26e27ac118e4" (UID: "e9de1723-3c15-4c53-8080-26e27ac118e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:51.251062 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.250835    4301 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/595ded14-5c69-427e-a232-e969122cf00f-home\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:51.251062 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.250859    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:51.251062 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.250866    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595ded14-5c69-427e-a232-e969122cf00f-tmp-dir" pod "595ded14-5c69-427e-a232-e969122cf00f" (UID: "595ded14-5c69-427e-a232-e969122cf00f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:51.251062 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.250874    4301 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-home\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:51.251062 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.250904    4301 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/595ded14-5c69-427e-a232-e969122cf00f-model-cache\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:51.251062 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.250919    4301 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-model-cache\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:51.251062 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.250933    4301 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd9c-f413-4852-869c-adc53259a977-model-cache\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:51.251062 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.250940    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/595ded14-5c69-427e-a232-e969122cf00f-tls-certs" pod "595ded14-5c69-427e-a232-e969122cf00f" (UID: "595ded14-5c69-427e-a232-e969122cf00f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:24:51.251062 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.250948    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:51.251062 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.250961    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9de1723-3c15-4c53-8080-26e27ac118e4-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:51.251062 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.250974    4301 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd9c-f413-4852-869c-adc53259a977-tmp-dir\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:51.251062 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.250987    4301 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca870469-c464-4272-a7d7-61081a3d2afc-tmp-dir\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:51.251798 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.251100    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-workload-certs" pod "e9de1723-3c15-4c53-8080-26e27ac118e4" (UID: "e9de1723-3c15-4c53-8080-26e27ac118e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:51.251798 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.251200    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa98bd9c-f413-4852-869c-adc53259a977-home" pod "fa98bd9c-f413-4852-869c-adc53259a977" (UID: "fa98bd9c-f413-4852-869c-adc53259a977"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:51.251798 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.251378    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-credential-socket" pod "e9de1723-3c15-4c53-8080-26e27ac118e4" (UID: "e9de1723-3c15-4c53-8080-26e27ac118e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:51.252519 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.252033    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-tmp-dir" pod "289e47fa-d3c8-49b4-aac7-0cd0959e64aa" (UID: "289e47fa-d3c8-49b4-aac7-0cd0959e64aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:51.252519 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.252207    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca870469-c464-4272-a7d7-61081a3d2afc-home" pod "ca870469-c464-4272-a7d7-61081a3d2afc" (UID: "ca870469-c464-4272-a7d7-61081a3d2afc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:51.252953 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.252922    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca870469-c464-4272-a7d7-61081a3d2afc-model-cache" pod "ca870469-c464-4272-a7d7-61081a3d2afc" (UID: "ca870469-c464-4272-a7d7-61081a3d2afc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:51.254650 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.254622    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/595ded14-5c69-427e-a232-e969122cf00f-kube-api-access-l74mg" pod "595ded14-5c69-427e-a232-e969122cf00f" (UID: "595ded14-5c69-427e-a232-e969122cf00f"). InnerVolumeSpecName "kube-api-access-l74mg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:51.255343 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.255308    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-tls-certs" pod "289e47fa-d3c8-49b4-aac7-0cd0959e64aa" (UID: "289e47fa-d3c8-49b4-aac7-0cd0959e64aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:24:51.255620 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.255595    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca870469-c464-4272-a7d7-61081a3d2afc-tls-certs" pod "ca870469-c464-4272-a7d7-61081a3d2afc" (UID: "ca870469-c464-4272-a7d7-61081a3d2afc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:24:51.256253 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.256227    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca870469-c464-4272-a7d7-61081a3d2afc-dshm" pod "ca870469-c464-4272-a7d7-61081a3d2afc" (UID: "ca870469-c464-4272-a7d7-61081a3d2afc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:51.256253 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.256227    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa98bd9c-f413-4852-869c-adc53259a977-kube-api-access-8rxtm" pod "fa98bd9c-f413-4852-869c-adc53259a977" (UID: "fa98bd9c-f413-4852-869c-adc53259a977"). InnerVolumeSpecName "kube-api-access-8rxtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:51.256685 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.256666    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa98bd9c-f413-4852-869c-adc53259a977-dshm" pod "fa98bd9c-f413-4852-869c-adc53259a977" (UID: "fa98bd9c-f413-4852-869c-adc53259a977"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:51.256972 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.256940    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9de1723-3c15-4c53-8080-26e27ac118e4-istio-token" pod "e9de1723-3c15-4c53-8080-26e27ac118e4" (UID: "e9de1723-3c15-4c53-8080-26e27ac118e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:51.257415 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.257389    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-kube-api-access-mx8t5" pod "289e47fa-d3c8-49b4-aac7-0cd0959e64aa" (UID: "289e47fa-d3c8-49b4-aac7-0cd0959e64aa"). InnerVolumeSpecName "kube-api-access-mx8t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:51.257706 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.257685    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9de1723-3c15-4c53-8080-26e27ac118e4-istio-podinfo" pod "e9de1723-3c15-4c53-8080-26e27ac118e4" (UID: "e9de1723-3c15-4c53-8080-26e27ac118e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:24:51.259455 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.259430    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-dshm" pod "289e47fa-d3c8-49b4-aac7-0cd0959e64aa" (UID: "289e47fa-d3c8-49b4-aac7-0cd0959e64aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:51.259556 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.259457    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-istio-envoy" pod "e9de1723-3c15-4c53-8080-26e27ac118e4" (UID: "e9de1723-3c15-4c53-8080-26e27ac118e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:51.259556 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.259539    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9de1723-3c15-4c53-8080-26e27ac118e4-kube-api-access-b8844" pod "e9de1723-3c15-4c53-8080-26e27ac118e4" (UID: "e9de1723-3c15-4c53-8080-26e27ac118e4"). InnerVolumeSpecName "kube-api-access-b8844". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:51.259667 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.259564    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa98bd9c-f413-4852-869c-adc53259a977-tls-certs" pod "fa98bd9c-f413-4852-869c-adc53259a977" (UID: "fa98bd9c-f413-4852-869c-adc53259a977"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:24:51.259667 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.259574    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca870469-c464-4272-a7d7-61081a3d2afc-kube-api-access-8vx7b" pod "ca870469-c464-4272-a7d7-61081a3d2afc" (UID: "ca870469-c464-4272-a7d7-61081a3d2afc"). InnerVolumeSpecName "kube-api-access-8vx7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:51.260771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.260751    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595ded14-5c69-427e-a232-e969122cf00f-dshm" pod "595ded14-5c69-427e-a232-e969122cf00f" (UID: "595ded14-5c69-427e-a232-e969122cf00f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:51.380771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.372443    4301 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd9c-f413-4852-869c-adc53259a977-dshm\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:51.380771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.372487    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:51.380771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.372503    4301 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca870469-c464-4272-a7d7-61081a3d2afc-model-cache\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:51.380771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.372534    4301 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa98bd9c-f413-4852-869c-adc53259a977-tls-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:51.380771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.372549    4301 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-dshm\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:51.380771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.372563    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8844\" (UniqueName: \"kubernetes.io/projected/e9de1723-3c15-4c53-8080-26e27ac118e4-kube-api-access-b8844\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:51.380771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.372578    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vx7b\" (UniqueName: \"kubernetes.io/projected/ca870469-c464-4272-a7d7-61081a3d2afc-kube-api-access-8vx7b\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:51.380771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.372592    4301 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-tmp-dir\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:51.380771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.372605    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:51.380771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.372617    4301 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/595ded14-5c69-427e-a232-e969122cf00f-tls-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:51.380771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.372632    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx8t5\" (UniqueName: \"kubernetes.io/projected/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-kube-api-access-mx8t5\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:51.380771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.372646    4301 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd9c-f413-4852-869c-adc53259a977-home\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:51.380771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.372659    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9de1723-3c15-4c53-8080-26e27ac118e4-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:51.380771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.372671    4301 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/595ded14-5c69-427e-a232-e969122cf00f-dshm\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:51.380771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.372690    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l74mg\" (UniqueName: \"kubernetes.io/projected/595ded14-5c69-427e-a232-e969122cf00f-kube-api-access-l74mg\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:51.380771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.372709    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9de1723-3c15-4c53-8080-26e27ac118e4-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:51.380771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.372724    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9de1723-3c15-4c53-8080-26e27ac118e4-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:51.380771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.372737    4301 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca870469-c464-4272-a7d7-61081a3d2afc-home\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:51.380771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.372750    4301 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/289e47fa-d3c8-49b4-aac7-0cd0959e64aa-tls-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:51.380771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.372762    4301 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca870469-c464-4272-a7d7-61081a3d2afc-tls-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:51.380771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.372776    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rxtm\" (UniqueName: \"kubernetes.io/projected/fa98bd9c-f413-4852-869c-adc53259a977-kube-api-access-8rxtm\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:51.380771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.372790    4301 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/595ded14-5c69-427e-a232-e969122cf00f-tmp-dir\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:51.380771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.372807    4301 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca870469-c464-4272-a7d7-61081a3d2afc-dshm\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:51.440789 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.440758    4301 generic.go:356] "Generic (PLEG): container finished" podID="595ded14-5c69-427e-a232-e969122cf00f" containerID="03db228446522bf212a08d53b5d2d1d60542e2a3767b08327929de6e374b0f3d" exitCode=0
Oct 31 23:24:51.440974 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.440866    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"
Oct 31 23:24:51.443062 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.443032    4301 generic.go:356] "Generic (PLEG): container finished" podID="e9de1723-3c15-4c53-8080-26e27ac118e4" containerID="192f6f07478d7ec10ff749b7697be9c1ba196660d01b7e03fb5f05cefc0cff0b" exitCode=0
Oct 31 23:24:51.443166 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.443100    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"
Oct 31 23:24:51.443826 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.443800    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b5f9bbb-897d-44dc-9eb8-5887091b4d64" path="/var/lib/kubelet/pods/2b5f9bbb-897d-44dc-9eb8-5887091b4d64/volumes"
Oct 31 23:24:51.444460 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.444437    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="394e1575-7171-45a2-8d19-ef551eaad0ea" path="/var/lib/kubelet/pods/394e1575-7171-45a2-8d19-ef551eaad0ea/volumes"
Oct 31 23:24:51.445048 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.445026    4301 generic.go:356] "Generic (PLEG): container finished" podID="ca870469-c464-4272-a7d7-61081a3d2afc" containerID="59f03b727cae67bed70d755db3db359e263bf6397d33b55041a696e1a0ed47b5" exitCode=0
Oct 31 23:24:51.445171 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.445140    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"
Oct 31 23:24:51.445239 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.445165    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bba885b3-4cbf-4677-bef9-94f492ca8794" path="/var/lib/kubelet/pods/bba885b3-4cbf-4677-bef9-94f492ca8794/volumes"
Oct 31 23:24:51.446130 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.446105    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" path="/var/lib/kubelet/pods/f2a4d961-adc7-4f38-9825-33f1230ddadb/volumes"
Oct 31 23:24:51.446613 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.446591    4301 generic.go:356] "Generic (PLEG): container finished" podID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerID="cd1a83d5f8ec000fbccaaa6a674ea6b1967c3ad6015a9c1c35dbd6f9b1dc9b55" exitCode=0
Oct 31 23:24:51.446741 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.446723    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"
Oct 31 23:24:51.447218 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.447193    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" event={"ID":"595ded14-5c69-427e-a232-e969122cf00f","Type":"ContainerDied","Data":"03db228446522bf212a08d53b5d2d1d60542e2a3767b08327929de6e374b0f3d"}
Oct 31 23:24:51.447298 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.447237    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg" event={"ID":"595ded14-5c69-427e-a232-e969122cf00f","Type":"ContainerDied","Data":"573b9ce04339a4a89e16c1ed87aa27918b0235bcf2f6b4758348056aa5d1e48b"}
Oct 31 23:24:51.447298 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.447255    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds" event={"ID":"e9de1723-3c15-4c53-8080-26e27ac118e4","Type":"ContainerDied","Data":"192f6f07478d7ec10ff749b7697be9c1ba196660d01b7e03fb5f05cefc0cff0b"}
Oct 31 23:24:51.447298 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.447285    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds" event={"ID":"e9de1723-3c15-4c53-8080-26e27ac118e4","Type":"ContainerDied","Data":"e3f8e7b62bec2e690418d58e011344ab9660e4fab49e0ec4fba877cd729bfd81"}
Oct 31 23:24:51.447500 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.447306    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" event={"ID":"ca870469-c464-4272-a7d7-61081a3d2afc","Type":"ContainerDied","Data":"59f03b727cae67bed70d755db3db359e263bf6397d33b55041a696e1a0ed47b5"}
Oct 31 23:24:51.447500 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.447321    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56" event={"ID":"ca870469-c464-4272-a7d7-61081a3d2afc","Type":"ContainerDied","Data":"74186068436c76f5f632ab84a4807374da41b81d49adc6c6f58094a4dc6948dc"}
Oct 31 23:24:51.447500 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.447335    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" event={"ID":"289e47fa-d3c8-49b4-aac7-0cd0959e64aa","Type":"ContainerDied","Data":"cd1a83d5f8ec000fbccaaa6a674ea6b1967c3ad6015a9c1c35dbd6f9b1dc9b55"}
Oct 31 23:24:51.447652 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.447351    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m" event={"ID":"289e47fa-d3c8-49b4-aac7-0cd0959e64aa","Type":"ContainerDied","Data":"27f5360a66a09b0abf979fbb6dacc0e3ca93f7911104fa151769e9877bb3175a"}
Oct 31 23:24:51.447652 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.447589    4301 scope.go:122] "RemoveContainer" containerID="03db228446522bf212a08d53b5d2d1d60542e2a3767b08327929de6e374b0f3d"
Oct 31 23:24:51.462792 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.461891    4301 generic.go:356] "Generic (PLEG): container finished" podID="fa98bd9c-f413-4852-869c-adc53259a977" containerID="aaeaff5fa1f733c19d17d6f7f7af6c9b749894159431c70fb2f4004c78cb9092" exitCode=0
Oct 31 23:24:51.462792 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.462014    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" event={"ID":"fa98bd9c-f413-4852-869c-adc53259a977","Type":"ContainerDied","Data":"aaeaff5fa1f733c19d17d6f7f7af6c9b749894159431c70fb2f4004c78cb9092"}
Oct 31 23:24:51.462792 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.462062    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9" event={"ID":"fa98bd9c-f413-4852-869c-adc53259a977","Type":"ContainerDied","Data":"c54889f9fe618bc9cea456b988366c175ec6d579af11d3f96ee37efd9f81a939"}
Oct 31 23:24:51.462792 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.462134    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"
Oct 31 23:24:51.473677 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.472797    4301 scope.go:122] "RemoveContainer" containerID="03db228446522bf212a08d53b5d2d1d60542e2a3767b08327929de6e374b0f3d"
Oct 31 23:24:51.473677 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:24:51.473173    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03db228446522bf212a08d53b5d2d1d60542e2a3767b08327929de6e374b0f3d\": container with ID starting with 03db228446522bf212a08d53b5d2d1d60542e2a3767b08327929de6e374b0f3d not found: ID does not exist" containerID="03db228446522bf212a08d53b5d2d1d60542e2a3767b08327929de6e374b0f3d"
Oct 31 23:24:51.473677 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.473200    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03db228446522bf212a08d53b5d2d1d60542e2a3767b08327929de6e374b0f3d"} err="failed to get container status \"03db228446522bf212a08d53b5d2d1d60542e2a3767b08327929de6e374b0f3d\": rpc error: code = NotFound desc = could not find container \"03db228446522bf212a08d53b5d2d1d60542e2a3767b08327929de6e374b0f3d\": container with ID starting with 03db228446522bf212a08d53b5d2d1d60542e2a3767b08327929de6e374b0f3d not found: ID does not exist"
Oct 31 23:24:51.473677 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.473217    4301 scope.go:122] "RemoveContainer" containerID="192f6f07478d7ec10ff749b7697be9c1ba196660d01b7e03fb5f05cefc0cff0b"
Oct 31 23:24:51.485237 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.485213    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"]
Oct 31 23:24:51.487605 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.487586    4301 scope.go:122] "RemoveContainer" containerID="b5a8fc8b2e0385dddba02178f1b292286a99189e348c99bf202d5738c63fbb6a"
Oct 31 23:24:51.489213 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.489190    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-bf279c-openshift-default-9d857b57c-k8kds"]
Oct 31 23:24:51.501497 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.501422    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"]
Oct 31 23:24:51.503172 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.503151    4301 scope.go:122] "RemoveContainer" containerID="192f6f07478d7ec10ff749b7697be9c1ba196660d01b7e03fb5f05cefc0cff0b"
Oct 31 23:24:51.507775 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.507743    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-135248/distinct-w4-135248-kserve-755d98c8b6-4zn56"]
Oct 31 23:24:51.512418 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:24:51.512325    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"192f6f07478d7ec10ff749b7697be9c1ba196660d01b7e03fb5f05cefc0cff0b\": container with ID starting with 192f6f07478d7ec10ff749b7697be9c1ba196660d01b7e03fb5f05cefc0cff0b not found: ID does not exist" containerID="192f6f07478d7ec10ff749b7697be9c1ba196660d01b7e03fb5f05cefc0cff0b"
Oct 31 23:24:51.512617 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.512587    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"192f6f07478d7ec10ff749b7697be9c1ba196660d01b7e03fb5f05cefc0cff0b"} err="failed to get container status \"192f6f07478d7ec10ff749b7697be9c1ba196660d01b7e03fb5f05cefc0cff0b\": rpc error: code = NotFound desc = could not find container \"192f6f07478d7ec10ff749b7697be9c1ba196660d01b7e03fb5f05cefc0cff0b\": container with ID starting with 192f6f07478d7ec10ff749b7697be9c1ba196660d01b7e03fb5f05cefc0cff0b not found: ID does not exist"
Oct 31 23:24:51.512717 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.512701    4301 scope.go:122] "RemoveContainer" containerID="b5a8fc8b2e0385dddba02178f1b292286a99189e348c99bf202d5738c63fbb6a"
Oct 31 23:24:51.513202 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:24:51.513172    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5a8fc8b2e0385dddba02178f1b292286a99189e348c99bf202d5738c63fbb6a\": container with ID starting with b5a8fc8b2e0385dddba02178f1b292286a99189e348c99bf202d5738c63fbb6a not found: ID does not exist" containerID="b5a8fc8b2e0385dddba02178f1b292286a99189e348c99bf202d5738c63fbb6a"
Oct 31 23:24:51.513643 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.513603    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5a8fc8b2e0385dddba02178f1b292286a99189e348c99bf202d5738c63fbb6a"} err="failed to get container status \"b5a8fc8b2e0385dddba02178f1b292286a99189e348c99bf202d5738c63fbb6a\": rpc error: code = NotFound desc = could not find container \"b5a8fc8b2e0385dddba02178f1b292286a99189e348c99bf202d5738c63fbb6a\": container with ID starting with b5a8fc8b2e0385dddba02178f1b292286a99189e348c99bf202d5738c63fbb6a not found: ID does not exist"
Oct 31 23:24:51.513643 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.513644    4301 scope.go:122] "RemoveContainer" containerID="59f03b727cae67bed70d755db3db359e263bf6397d33b55041a696e1a0ed47b5"
Oct 31 23:24:51.516417 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.516330    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"]
Oct 31 23:24:51.519500 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.519478    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-135248/premium-w4-135248-kserve-6474f475c5-lxh6m"]
Oct 31 23:24:51.523979 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.523959    4301 scope.go:122] "RemoveContainer" containerID="59f03b727cae67bed70d755db3db359e263bf6397d33b55041a696e1a0ed47b5"
Oct 31 23:24:51.524263 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:24:51.524243    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59f03b727cae67bed70d755db3db359e263bf6397d33b55041a696e1a0ed47b5\": container with ID starting with 59f03b727cae67bed70d755db3db359e263bf6397d33b55041a696e1a0ed47b5 not found: ID does not exist" containerID="59f03b727cae67bed70d755db3db359e263bf6397d33b55041a696e1a0ed47b5"
Oct 31 23:24:51.524320 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.524272    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59f03b727cae67bed70d755db3db359e263bf6397d33b55041a696e1a0ed47b5"} err="failed to get container status \"59f03b727cae67bed70d755db3db359e263bf6397d33b55041a696e1a0ed47b5\": rpc error: code = NotFound desc = could not find container \"59f03b727cae67bed70d755db3db359e263bf6397d33b55041a696e1a0ed47b5\": container with ID starting with 59f03b727cae67bed70d755db3db359e263bf6397d33b55041a696e1a0ed47b5 not found: ID does not exist"
Oct 31 23:24:51.524320 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.524289    4301 scope.go:122] "RemoveContainer" containerID="cd1a83d5f8ec000fbccaaa6a674ea6b1967c3ad6015a9c1c35dbd6f9b1dc9b55"
Oct 31 23:24:51.528319 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.528301    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"]
Oct 31 23:24:51.531922 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.531900    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-135248/simulator-w4-135248-kserve-6b9dcfcd66-t29w9"]
Oct 31 23:24:51.534943 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.534926    4301 scope.go:122] "RemoveContainer" containerID="cd1a83d5f8ec000fbccaaa6a674ea6b1967c3ad6015a9c1c35dbd6f9b1dc9b55"
Oct 31 23:24:51.535218 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:24:51.535201    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd1a83d5f8ec000fbccaaa6a674ea6b1967c3ad6015a9c1c35dbd6f9b1dc9b55\": container with ID starting with cd1a83d5f8ec000fbccaaa6a674ea6b1967c3ad6015a9c1c35dbd6f9b1dc9b55 not found: ID does not exist" containerID="cd1a83d5f8ec000fbccaaa6a674ea6b1967c3ad6015a9c1c35dbd6f9b1dc9b55"
Oct 31 23:24:51.535271 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.535225    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd1a83d5f8ec000fbccaaa6a674ea6b1967c3ad6015a9c1c35dbd6f9b1dc9b55"} err="failed to get container status \"cd1a83d5f8ec000fbccaaa6a674ea6b1967c3ad6015a9c1c35dbd6f9b1dc9b55\": rpc error: code = NotFound desc = could not find container \"cd1a83d5f8ec000fbccaaa6a674ea6b1967c3ad6015a9c1c35dbd6f9b1dc9b55\": container with ID starting with cd1a83d5f8ec000fbccaaa6a674ea6b1967c3ad6015a9c1c35dbd6f9b1dc9b55 not found: ID does not exist"
Oct 31 23:24:51.535271 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.535241    4301 scope.go:122] "RemoveContainer" containerID="aaeaff5fa1f733c19d17d6f7f7af6c9b749894159431c70fb2f4004c78cb9092"
Oct 31 23:24:51.544081 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.544056    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"]
Oct 31 23:24:51.544737 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.544719    4301 scope.go:122] "RemoveContainer" containerID="aaeaff5fa1f733c19d17d6f7f7af6c9b749894159431c70fb2f4004c78cb9092"
Oct 31 23:24:51.545135 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:24:51.545109    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaeaff5fa1f733c19d17d6f7f7af6c9b749894159431c70fb2f4004c78cb9092\": container with ID starting with aaeaff5fa1f733c19d17d6f7f7af6c9b749894159431c70fb2f4004c78cb9092 not found: ID does not exist" containerID="aaeaff5fa1f733c19d17d6f7f7af6c9b749894159431c70fb2f4004c78cb9092"
Oct 31 23:24:51.545260 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.545144    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaeaff5fa1f733c19d17d6f7f7af6c9b749894159431c70fb2f4004c78cb9092"} err="failed to get container status \"aaeaff5fa1f733c19d17d6f7f7af6c9b749894159431c70fb2f4004c78cb9092\": rpc error: code = NotFound desc = could not find container \"aaeaff5fa1f733c19d17d6f7f7af6c9b749894159431c70fb2f4004c78cb9092\": container with ID starting with aaeaff5fa1f733c19d17d6f7f7af6c9b749894159431c70fb2f4004c78cb9092 not found: ID does not exist"
Oct 31 23:24:51.548074 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:51.548053    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-135248/distinct-2-w4-135248-kserve-74fc5794d9-wjdzg"]
Oct 31 23:24:52.906897 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:52.906856    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d7eed-7c97f4ff9f-whnrt" podUID="a3bc4bb3-e967-4fe3-a6b9-71377f8f6047" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 23:24:53.403136 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:53.403105    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" path="/var/lib/kubelet/pods/289e47fa-d3c8-49b4-aac7-0cd0959e64aa/volumes"
Oct 31 23:24:53.403561 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:53.403546    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="595ded14-5c69-427e-a232-e969122cf00f" path="/var/lib/kubelet/pods/595ded14-5c69-427e-a232-e969122cf00f/volumes"
Oct 31 23:24:53.403940 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:53.403927    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" path="/var/lib/kubelet/pods/ca870469-c464-4272-a7d7-61081a3d2afc/volumes"
Oct 31 23:24:53.404510 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:53.404491    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9de1723-3c15-4c53-8080-26e27ac118e4" path="/var/lib/kubelet/pods/e9de1723-3c15-4c53-8080-26e27ac118e4/volumes"
Oct 31 23:24:53.404983 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:53.404969    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa98bd9c-f413-4852-869c-adc53259a977" path="/var/lib/kubelet/pods/fa98bd9c-f413-4852-869c-adc53259a977/volumes"
Oct 31 23:24:56.299198 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.298125    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"]
Oct 31 23:24:56.407432 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.407131    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:24:56.501697 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.501653    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istiod-ca-cert\") pod \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") "
Oct 31 23:24:56.501885 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.501719    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istio-data\") pod \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") "
Oct 31 23:24:56.501885 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.501776    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-workload-socket\") pod \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") "
Oct 31 23:24:56.501885 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.501817    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-workload-certs\") pod \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") "
Oct 31 23:24:56.501885 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.501866    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istio-token\" (UniqueName: \"kubernetes.io/projected/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istio-token\") pod \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") "
Oct 31 23:24:56.502131 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.501891    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a58ab83-2394-48a8-aa4f-04c22ee67a24-kube-api-access-8pg97\" (UniqueName: \"kubernetes.io/projected/9a58ab83-2394-48a8-aa4f-04c22ee67a24-kube-api-access-8pg97\") pod \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") "
Oct 31 23:24:56.502131 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.501920    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istio-podinfo\") pod \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") "
Oct 31 23:24:56.502131 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.501963    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-credential-socket\") pod \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") "
Oct 31 23:24:56.502131 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.502032    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istiod-ca-cert" pod "9a58ab83-2394-48a8-aa4f-04c22ee67a24" (UID: "9a58ab83-2394-48a8-aa4f-04c22ee67a24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:24:56.502131 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.502068    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istio-data" pod "9a58ab83-2394-48a8-aa4f-04c22ee67a24" (UID: "9a58ab83-2394-48a8-aa4f-04c22ee67a24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:56.502131 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.502085    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istio-envoy\") pod \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\" (UID: \"9a58ab83-2394-48a8-aa4f-04c22ee67a24\") "
Oct 31 23:24:56.502728 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.502706    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:56.502728 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.502728    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:56.503691 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.503671    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-workload-socket" pod "9a58ab83-2394-48a8-aa4f-04c22ee67a24" (UID: "9a58ab83-2394-48a8-aa4f-04c22ee67a24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:56.503910 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.503892    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-credential-socket" pod "9a58ab83-2394-48a8-aa4f-04c22ee67a24" (UID: "9a58ab83-2394-48a8-aa4f-04c22ee67a24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:56.504111 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.504076    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-workload-certs" pod "9a58ab83-2394-48a8-aa4f-04c22ee67a24" (UID: "9a58ab83-2394-48a8-aa4f-04c22ee67a24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:56.555481 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.551545    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istio-podinfo" pod "9a58ab83-2394-48a8-aa4f-04c22ee67a24" (UID: "9a58ab83-2394-48a8-aa4f-04c22ee67a24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:24:56.570423 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.565984    4301 generic.go:356] "Generic (PLEG): container finished" podID="9a58ab83-2394-48a8-aa4f-04c22ee67a24" containerID="0b88e034f850cc37100ede8de632d0db0f238582eb0c3975b27dea1b6a615860" exitCode=0
Oct 31 23:24:56.570423 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.566122    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx" event={"ID":"9a58ab83-2394-48a8-aa4f-04c22ee67a24","Type":"ContainerDied","Data":"0b88e034f850cc37100ede8de632d0db0f238582eb0c3975b27dea1b6a615860"}
Oct 31 23:24:56.570423 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.566175    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx" event={"ID":"9a58ab83-2394-48a8-aa4f-04c22ee67a24","Type":"ContainerDied","Data":"2de3b9cb2acb2f6b7d0bacaf1fe3c9cbd480f6f9d1d1b27d0e8f5668b4ca0b1f"}
Oct 31 23:24:56.570423 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.566204    4301 scope.go:122] "RemoveContainer" containerID="0b88e034f850cc37100ede8de632d0db0f238582eb0c3975b27dea1b6a615860"
Oct 31 23:24:56.570423 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.566332    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"
Oct 31 23:24:56.592382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.589840    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istio-token" pod "9a58ab83-2394-48a8-aa4f-04c22ee67a24" (UID: "9a58ab83-2394-48a8-aa4f-04c22ee67a24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:56.593315 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.593265    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a58ab83-2394-48a8-aa4f-04c22ee67a24-kube-api-access-8pg97" pod "9a58ab83-2394-48a8-aa4f-04c22ee67a24" (UID: "9a58ab83-2394-48a8-aa4f-04c22ee67a24"). InnerVolumeSpecName "kube-api-access-8pg97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:56.595799 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.595774    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istio-envoy" pod "9a58ab83-2394-48a8-aa4f-04c22ee67a24" (UID: "9a58ab83-2394-48a8-aa4f-04c22ee67a24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:24:56.615787 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.615652    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:56.615787 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.615688    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:56.615787 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.615702    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:56.615787 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.615715    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pg97\" (UniqueName: \"kubernetes.io/projected/9a58ab83-2394-48a8-aa4f-04c22ee67a24-kube-api-access-8pg97\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:56.615787 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.615731    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:56.615787 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.615747    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:56.615787 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.615760    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a58ab83-2394-48a8-aa4f-04c22ee67a24-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:56.784845 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.783013    4301 scope.go:122] "RemoveContainer" containerID="d48357bb04b7e1e03a059677a5ad42bbccdf1017aba1ca95dad92fa740178e30"
Oct 31 23:24:56.808824 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.808438    4301 scope.go:122] "RemoveContainer" containerID="0b88e034f850cc37100ede8de632d0db0f238582eb0c3975b27dea1b6a615860"
Oct 31 23:24:56.815673 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:24:56.809139    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b88e034f850cc37100ede8de632d0db0f238582eb0c3975b27dea1b6a615860\": container with ID starting with 0b88e034f850cc37100ede8de632d0db0f238582eb0c3975b27dea1b6a615860 not found: ID does not exist" containerID="0b88e034f850cc37100ede8de632d0db0f238582eb0c3975b27dea1b6a615860"
Oct 31 23:24:56.815673 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.809190    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b88e034f850cc37100ede8de632d0db0f238582eb0c3975b27dea1b6a615860"} err="failed to get container status \"0b88e034f850cc37100ede8de632d0db0f238582eb0c3975b27dea1b6a615860\": rpc error: code = NotFound desc = could not find container \"0b88e034f850cc37100ede8de632d0db0f238582eb0c3975b27dea1b6a615860\": container with ID starting with 0b88e034f850cc37100ede8de632d0db0f238582eb0c3975b27dea1b6a615860 not found: ID does not exist"
Oct 31 23:24:56.815673 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.809215    4301 scope.go:122] "RemoveContainer" containerID="d48357bb04b7e1e03a059677a5ad42bbccdf1017aba1ca95dad92fa740178e30"
Oct 31 23:24:56.816265 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:24:56.816176    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d48357bb04b7e1e03a059677a5ad42bbccdf1017aba1ca95dad92fa740178e30\": container with ID starting with d48357bb04b7e1e03a059677a5ad42bbccdf1017aba1ca95dad92fa740178e30 not found: ID does not exist" containerID="d48357bb04b7e1e03a059677a5ad42bbccdf1017aba1ca95dad92fa740178e30"
Oct 31 23:24:56.816265 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.816216    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d48357bb04b7e1e03a059677a5ad42bbccdf1017aba1ca95dad92fa740178e30"} err="failed to get container status \"d48357bb04b7e1e03a059677a5ad42bbccdf1017aba1ca95dad92fa740178e30\": rpc error: code = NotFound desc = could not find container \"d48357bb04b7e1e03a059677a5ad42bbccdf1017aba1ca95dad92fa740178e30\": container with ID starting with d48357bb04b7e1e03a059677a5ad42bbccdf1017aba1ca95dad92fa740178e30 not found: ID does not exist"
Oct 31 23:24:56.898226 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.898194    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"]
Oct 31 23:24:56.920471 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:56.920440    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7c4c91-openshift-default-6586758b7c-t44cx"]
Oct 31 23:24:57.431383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:57.424340    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a58ab83-2394-48a8-aa4f-04c22ee67a24" path="/var/lib/kubelet/pods/9a58ab83-2394-48a8-aa4f-04c22ee67a24/volumes"
Oct 31 23:24:58.646830 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:58.646795    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg"]
Oct 31 23:24:58.647696 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:58.647662    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg" podUID="7ee9b5ef-cded-4849-8ebc-86c305c4e8a7" containerName="maas-api" containerID="cri-o://d36b7c285acd572606a5d13337f21d5cc35bddd95af24fc52b9474678913f385" gracePeriod=30
Oct 31 23:24:59.086380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:59.076201    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg"
Oct 31 23:24:59.178305 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:59.178264    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ee9b5ef-cded-4849-8ebc-86c305c4e8a7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ee9b5ef-cded-4849-8ebc-86c305c4e8a7-metrics-certs\") pod \"7ee9b5ef-cded-4849-8ebc-86c305c4e8a7\" (UID: \"7ee9b5ef-cded-4849-8ebc-86c305c4e8a7\") "
Oct 31 23:24:59.178533 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:59.178473    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ee9b5ef-cded-4849-8ebc-86c305c4e8a7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee9b5ef-cded-4849-8ebc-86c305c4e8a7-openshift-service-ca\") pod \"7ee9b5ef-cded-4849-8ebc-86c305c4e8a7\" (UID: \"7ee9b5ef-cded-4849-8ebc-86c305c4e8a7\") "
Oct 31 23:24:59.178787 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:59.178738    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee9b5ef-cded-4849-8ebc-86c305c4e8a7-kube-api-access-t7pdn\" (UniqueName: \"kubernetes.io/projected/7ee9b5ef-cded-4849-8ebc-86c305c4e8a7-kube-api-access-t7pdn\") pod \"7ee9b5ef-cded-4849-8ebc-86c305c4e8a7\" (UID: \"7ee9b5ef-cded-4849-8ebc-86c305c4e8a7\") "
Oct 31 23:24:59.178904 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:59.178805    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ee9b5ef-cded-4849-8ebc-86c305c4e8a7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ee9b5ef-cded-4849-8ebc-86c305c4e8a7-maas-api-tls\") pod \"7ee9b5ef-cded-4849-8ebc-86c305c4e8a7\" (UID: \"7ee9b5ef-cded-4849-8ebc-86c305c4e8a7\") "
Oct 31 23:24:59.195396 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:59.180719    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee9b5ef-cded-4849-8ebc-86c305c4e8a7-openshift-service-ca" pod "7ee9b5ef-cded-4849-8ebc-86c305c4e8a7" (UID: "7ee9b5ef-cded-4849-8ebc-86c305c4e8a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:24:59.195396 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:59.183335    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ee9b5ef-cded-4849-8ebc-86c305c4e8a7-metrics-certs" pod "7ee9b5ef-cded-4849-8ebc-86c305c4e8a7" (UID: "7ee9b5ef-cded-4849-8ebc-86c305c4e8a7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:24:59.195396 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:59.184946    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ee9b5ef-cded-4849-8ebc-86c305c4e8a7-maas-api-tls" pod "7ee9b5ef-cded-4849-8ebc-86c305c4e8a7" (UID: "7ee9b5ef-cded-4849-8ebc-86c305c4e8a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:24:59.195396 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:59.186245    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee9b5ef-cded-4849-8ebc-86c305c4e8a7-kube-api-access-t7pdn" pod "7ee9b5ef-cded-4849-8ebc-86c305c4e8a7" (UID: "7ee9b5ef-cded-4849-8ebc-86c305c4e8a7"). InnerVolumeSpecName "kube-api-access-t7pdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:24:59.281277 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:59.281198    4301 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee9b5ef-cded-4849-8ebc-86c305c4e8a7-openshift-service-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:59.281277 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:59.281237    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7pdn\" (UniqueName: \"kubernetes.io/projected/7ee9b5ef-cded-4849-8ebc-86c305c4e8a7-kube-api-access-t7pdn\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:59.281277 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:59.281252    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ee9b5ef-cded-4849-8ebc-86c305c4e8a7-maas-api-tls\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:59.281277 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:59.281266    4301 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ee9b5ef-cded-4849-8ebc-86c305c4e8a7-metrics-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:24:59.670029 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:59.669993    4301 generic.go:356] "Generic (PLEG): container finished" podID="7ee9b5ef-cded-4849-8ebc-86c305c4e8a7" containerID="d36b7c285acd572606a5d13337f21d5cc35bddd95af24fc52b9474678913f385" exitCode=0
Oct 31 23:24:59.670457 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:59.670035    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg"
Oct 31 23:24:59.670457 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:59.670075    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg" event={"ID":"7ee9b5ef-cded-4849-8ebc-86c305c4e8a7","Type":"ContainerDied","Data":"d36b7c285acd572606a5d13337f21d5cc35bddd95af24fc52b9474678913f385"}
Oct 31 23:24:59.670457 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:59.670134    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg" event={"ID":"7ee9b5ef-cded-4849-8ebc-86c305c4e8a7","Type":"ContainerDied","Data":"2568db3705e7da8cb0159e76ccce3ef1e3b7e29d5d6044bfc246bd4c49bf9bcf"}
Oct 31 23:24:59.670457 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:59.670157    4301 scope.go:122] "RemoveContainer" containerID="d36b7c285acd572606a5d13337f21d5cc35bddd95af24fc52b9474678913f385"
Oct 31 23:24:59.682809 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:59.682781    4301 scope.go:122] "RemoveContainer" containerID="d36b7c285acd572606a5d13337f21d5cc35bddd95af24fc52b9474678913f385"
Oct 31 23:24:59.683115 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:24:59.683098    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d36b7c285acd572606a5d13337f21d5cc35bddd95af24fc52b9474678913f385\": container with ID starting with d36b7c285acd572606a5d13337f21d5cc35bddd95af24fc52b9474678913f385 not found: ID does not exist" containerID="d36b7c285acd572606a5d13337f21d5cc35bddd95af24fc52b9474678913f385"
Oct 31 23:24:59.683169 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:59.683125    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d36b7c285acd572606a5d13337f21d5cc35bddd95af24fc52b9474678913f385"} err="failed to get container status \"d36b7c285acd572606a5d13337f21d5cc35bddd95af24fc52b9474678913f385\": rpc error: code = NotFound desc = could not find container \"d36b7c285acd572606a5d13337f21d5cc35bddd95af24fc52b9474678913f385\": container with ID starting with d36b7c285acd572606a5d13337f21d5cc35bddd95af24fc52b9474678913f385 not found: ID does not exist"
Oct 31 23:24:59.688616 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:59.688595    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg"]
Oct 31 23:24:59.697924 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:24:59.697163    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d7eed-7d85475bd5-8xqbg"]
Oct 31 23:25:01.403712 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.403682    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ee9b5ef-cded-4849-8ebc-86c305c4e8a7" path="/var/lib/kubelet/pods/7ee9b5ef-cded-4849-8ebc-86c305c4e8a7/volumes"
Oct 31 23:25:01.713527 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.713426    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f98a428-bzspr"]
Oct 31 23:25:01.720967 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.720921    4301 cpu_manager.go:414] "containerMap: removing container" podUID="bba885b3-4cbf-4677-bef9-94f492ca8794" containerName="istio-proxy"
Oct 31 23:25:01.721282 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.721254    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bba885b3-4cbf-4677-bef9-94f492ca8794" containerName="istio-proxy"
Oct 31 23:25:01.721445 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.721430    4301 cpu_manager.go:414] "containerMap: removing container" podUID="394e1575-7171-45a2-8d19-ef551eaad0ea" containerName="istio-proxy"
Oct 31 23:25:01.721538 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.721527    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="394e1575-7171-45a2-8d19-ef551eaad0ea" containerName="istio-proxy"
Oct 31 23:25:01.721626 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.721616    4301 cpu_manager.go:414] "containerMap: removing container" podUID="394e1575-7171-45a2-8d19-ef551eaad0ea" containerName="istio-proxy"
Oct 31 23:25:01.721708 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.721699    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="394e1575-7171-45a2-8d19-ef551eaad0ea" containerName="istio-proxy"
Oct 31 23:25:01.721776 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.721768    4301 cpu_manager.go:414] "containerMap: removing container" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main"
Oct 31 23:25:01.721857 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.721844    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main"
Oct 31 23:25:01.721943 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.721935    4301 cpu_manager.go:414] "containerMap: removing container" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main"
Oct 31 23:25:01.722007 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.721998    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main"
Oct 31 23:25:01.722085 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.722074    4301 cpu_manager.go:414] "containerMap: removing container" podUID="e9de1723-3c15-4c53-8080-26e27ac118e4" containerName="istio-proxy"
Oct 31 23:25:01.722242 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.722224    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9de1723-3c15-4c53-8080-26e27ac118e4" containerName="istio-proxy"
Oct 31 23:25:01.722393 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.722379    4301 cpu_manager.go:414] "containerMap: removing container" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main"
Oct 31 23:25:01.722518 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.722506    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main"
Oct 31 23:25:01.722952 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.722932    4301 cpu_manager.go:414] "containerMap: removing container" podUID="9760fd20-c563-4f5a-be5a-eeca1a7898a3" containerName="revoke-keys"
Oct 31 23:25:01.723071 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.723055    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9760fd20-c563-4f5a-be5a-eeca1a7898a3" containerName="revoke-keys"
Oct 31 23:25:01.723175 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.723165    4301 cpu_manager.go:414] "containerMap: removing container" podUID="84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3" containerName="istio-proxy"
Oct 31 23:25:01.723254 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.723242    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3" containerName="istio-proxy"
Oct 31 23:25:01.723348 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.723337    4301 cpu_manager.go:414] "containerMap: removing container" podUID="929c3031-62b4-4587-9f53-75de251130c8" containerName="istio-proxy"
Oct 31 23:25:01.723516 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.723501    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="929c3031-62b4-4587-9f53-75de251130c8" containerName="istio-proxy"
Oct 31 23:25:01.723732 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.723718    4301 cpu_manager.go:414] "containerMap: removing container" podUID="9a58ab83-2394-48a8-aa4f-04c22ee67a24" containerName="istio-proxy"
Oct 31 23:25:01.723823 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.723812    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a58ab83-2394-48a8-aa4f-04c22ee67a24" containerName="istio-proxy"
Oct 31 23:25:01.723912 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.723903    4301 cpu_manager.go:414] "containerMap: removing container" podUID="2b5f9bbb-897d-44dc-9eb8-5887091b4d64" containerName="istio-proxy"
Oct 31 23:25:01.723984 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.723974    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b5f9bbb-897d-44dc-9eb8-5887091b4d64" containerName="istio-proxy"
Oct 31 23:25:01.724052 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.724044    4301 cpu_manager.go:414] "containerMap: removing container" podUID="7ee9b5ef-cded-4849-8ebc-86c305c4e8a7" containerName="maas-api"
Oct 31 23:25:01.724121 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.724112    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ee9b5ef-cded-4849-8ebc-86c305c4e8a7" containerName="maas-api"
Oct 31 23:25:01.724197 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.724189    4301 cpu_manager.go:414] "containerMap: removing container" podUID="9a58ab83-2394-48a8-aa4f-04c22ee67a24" containerName="istio-proxy"
Oct 31 23:25:01.724267 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.724258    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a58ab83-2394-48a8-aa4f-04c22ee67a24" containerName="istio-proxy"
Oct 31 23:25:01.724339 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.724330    4301 cpu_manager.go:414] "containerMap: removing container" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main"
Oct 31 23:25:01.724465 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.724453    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main"
Oct 31 23:25:01.724548 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.724539    4301 cpu_manager.go:414] "containerMap: removing container" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main"
Oct 31 23:25:01.724620 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.724610    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main"
Oct 31 23:25:01.724695 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.724686    4301 cpu_manager.go:414] "containerMap: removing container" podUID="19145c81-9796-4ca8-92b4-8098589f845e" containerName="istio-proxy"
Oct 31 23:25:01.724776 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.724766    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19145c81-9796-4ca8-92b4-8098589f845e" containerName="istio-proxy"
Oct 31 23:25:01.724855 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.724846    4301 cpu_manager.go:414] "containerMap: removing container" podUID="e9de1723-3c15-4c53-8080-26e27ac118e4" containerName="istio-proxy"
Oct 31 23:25:01.724923 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.724913    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9de1723-3c15-4c53-8080-26e27ac118e4" containerName="istio-proxy"
Oct 31 23:25:01.725143 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.725130    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b5f9bbb-897d-44dc-9eb8-5887091b4d64" containerName="istio-proxy"
Oct 31 23:25:01.725249 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.725240    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ee9b5ef-cded-4849-8ebc-86c305c4e8a7" containerName="maas-api"
Oct 31 23:25:01.725340 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.725330    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a58ab83-2394-48a8-aa4f-04c22ee67a24" containerName="istio-proxy"
Oct 31 23:25:01.725464 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.725452    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa98bd9c-f413-4852-869c-adc53259a977" containerName="main"
Oct 31 23:25:01.725544 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.725534    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="289e47fa-d3c8-49b4-aac7-0cd0959e64aa" containerName="main"
Oct 31 23:25:01.725636 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.725614    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="19145c81-9796-4ca8-92b4-8098589f845e" containerName="istio-proxy"
Oct 31 23:25:01.725723 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.725707    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9de1723-3c15-4c53-8080-26e27ac118e4" containerName="istio-proxy"
Oct 31 23:25:01.725811 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.725726    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="bba885b3-4cbf-4677-bef9-94f492ca8794" containerName="istio-proxy"
Oct 31 23:25:01.725811 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.725740    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="394e1575-7171-45a2-8d19-ef551eaad0ea" containerName="istio-proxy"
Oct 31 23:25:01.725811 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.725750    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="394e1575-7171-45a2-8d19-ef551eaad0ea" containerName="istio-proxy"
Oct 31 23:25:01.725811 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.725759    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="595ded14-5c69-427e-a232-e969122cf00f" containerName="main"
Oct 31 23:25:01.725811 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.725769    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca870469-c464-4272-a7d7-61081a3d2afc" containerName="main"
Oct 31 23:25:01.725811 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.725778    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="46881476-3103-4ee9-9b90-ec0ba9f30c3e" containerName="istio-proxy"
Oct 31 23:25:01.725811 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.725789    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9de1723-3c15-4c53-8080-26e27ac118e4" containerName="istio-proxy"
Oct 31 23:25:01.725811 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.725798    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="9760fd20-c563-4f5a-be5a-eeca1a7898a3" containerName="revoke-keys"
Oct 31 23:25:01.725811 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.725809    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2a4d961-adc7-4f38-9825-33f1230ddadb" containerName="main"
Oct 31 23:25:01.726216 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.725820    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="84b0a9a3-a72d-4a7e-8b45-db92fe46a6f3" containerName="istio-proxy"
Oct 31 23:25:01.726216 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.725827    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="929c3031-62b4-4587-9f53-75de251130c8" containerName="istio-proxy"
Oct 31 23:25:01.744045 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.744013    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f98a428-bzspr"
Oct 31 23:25:01.750240 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.750203    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 23:25:01.750465 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.750287    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\""
Oct 31 23:25:01.847189 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.847144    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fbc8cf6-9cce-414c-9146-8a2ebd840fe0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f98a428-bzspr\" (UID: \"2fbc8cf6-9cce-414c-9146-8a2ebd840fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f98a428-bzspr"
Oct 31 23:25:01.873553 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.873517    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pflfk\" (UniqueName: \"kubernetes.io/projected/2fbc8cf6-9cce-414c-9146-8a2ebd840fe0-kube-api-access-pflfk\") pod \"maas-api-revoke-sub-0f98a428-bzspr\" (UID: \"2fbc8cf6-9cce-414c-9146-8a2ebd840fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f98a428-bzspr"
Oct 31 23:25:01.881631 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.881583    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fbc8cf6-9cce-414c-9146-8a2ebd840fe0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f98a428-bzspr\" (UID: \"2fbc8cf6-9cce-414c-9146-8a2ebd840fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f98a428-bzspr"
Oct 31 23:25:01.911379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.907460    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f98a428-bzspr"]
Oct 31 23:25:01.984446 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.984005    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fbc8cf6-9cce-414c-9146-8a2ebd840fe0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f98a428-bzspr\" (UID: \"2fbc8cf6-9cce-414c-9146-8a2ebd840fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f98a428-bzspr"
Oct 31 23:25:01.984446 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.984090    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pflfk\" (UniqueName: \"kubernetes.io/projected/2fbc8cf6-9cce-414c-9146-8a2ebd840fe0-kube-api-access-pflfk\") pod \"maas-api-revoke-sub-0f98a428-bzspr\" (UID: \"2fbc8cf6-9cce-414c-9146-8a2ebd840fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f98a428-bzspr"
Oct 31 23:25:01.984446 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.984128    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fbc8cf6-9cce-414c-9146-8a2ebd840fe0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f98a428-bzspr\" (UID: \"2fbc8cf6-9cce-414c-9146-8a2ebd840fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f98a428-bzspr"
Oct 31 23:25:01.989477 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:01.986672    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fbc8cf6-9cce-414c-9146-8a2ebd840fe0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f98a428-bzspr\" (UID: \"2fbc8cf6-9cce-414c-9146-8a2ebd840fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f98a428-bzspr"
Oct 31 23:25:02.003585 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:02.003550    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pflfk\" (UniqueName: \"kubernetes.io/projected/2fbc8cf6-9cce-414c-9146-8a2ebd840fe0-kube-api-access-pflfk\") pod \"maas-api-revoke-sub-0f98a428-bzspr\" (UID: \"2fbc8cf6-9cce-414c-9146-8a2ebd840fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f98a428-bzspr"
Oct 31 23:25:02.005337 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:02.005019    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fbc8cf6-9cce-414c-9146-8a2ebd840fe0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f98a428-bzspr\" (UID: \"2fbc8cf6-9cce-414c-9146-8a2ebd840fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f98a428-bzspr"
Oct 31 23:25:02.186054 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:02.185719    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f98a428-bzspr"
Oct 31 23:25:02.536090 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:02.536015    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f98a428-bzspr"]
Oct 31 23:25:02.539944 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:25:02.539918    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fbc8cf6_9cce_414c_9146_8a2ebd840fe0.slice/crio-73ae308343dc093ded53301d85c24f740de9e5662604640d6283fc3e83029d2c WatchSource:0}: Error finding container 73ae308343dc093ded53301d85c24f740de9e5662604640d6283fc3e83029d2c: Status 404 returned error can't find the container with id 73ae308343dc093ded53301d85c24f740de9e5662604640d6283fc3e83029d2c
Oct 31 23:25:02.554006 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:25:02.551206    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:25:02.693836 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:02.693802    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f98a428-bzspr" event={"ID":"2fbc8cf6-9cce-414c-9146-8a2ebd840fe0","Type":"ContainerStarted","Data":"73ae308343dc093ded53301d85c24f740de9e5662604640d6283fc3e83029d2c"}
Oct 31 23:25:02.962574 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:02.962520    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d7eed-7c97f4ff9f-whnrt" podUID="a3bc4bb3-e967-4fe3-a6b9-71377f8f6047" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 23:25:03.701689 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:03.701658    4301 generic.go:356] "Generic (PLEG): container finished" podID="2fbc8cf6-9cce-414c-9146-8a2ebd840fe0" containerID="8446f14bd9720a8a26f59e60cf422612fabed9660df53d97854b34a4f614203b" exitCode=0
Oct 31 23:25:03.702105 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:03.701743    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f98a428-bzspr" event={"ID":"2fbc8cf6-9cce-414c-9146-8a2ebd840fe0","Type":"ContainerDied","Data":"8446f14bd9720a8a26f59e60cf422612fabed9660df53d97854b34a4f614203b"}
Oct 31 23:25:04.040781 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.040692    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main" containerID="cri-o://04ed85ed4e4daae0bc0bd6980041d7768a9d0d8b4a00a987e38724cc694e38fb" gracePeriod=45
Oct 31 23:25:04.061988 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.061943    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" containerID="cri-o://d13f078c8c0a8d358fc427ef0605270c01dc7a5937327089a1f781d81d4645b5" gracePeriod=45
Oct 31 23:25:04.162815 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.162780    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": dial tcp 10.133.0.38:8000: connect: connection refused"
Oct 31 23:25:04.440850 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.440691    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:25:04.442907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.442880    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:25:04.617282 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.617201    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1486aeca-b15c-4611-beda-0baf2e503a89-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1486aeca-b15c-4611-beda-0baf2e503a89-tmp-dir\") pod \"1486aeca-b15c-4611-beda-0baf2e503a89\" (UID: \"1486aeca-b15c-4611-beda-0baf2e503a89\") "
Oct 31 23:25:04.617282 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.617252    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9a165f7-116b-4c15-9836-349bebeab135-tls-certs\" (UniqueName: \"kubernetes.io/secret/d9a165f7-116b-4c15-9836-349bebeab135-tls-certs\") pod \"d9a165f7-116b-4c15-9836-349bebeab135\" (UID: \"d9a165f7-116b-4c15-9836-349bebeab135\") "
Oct 31 23:25:04.617282 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.617282    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9a165f7-116b-4c15-9836-349bebeab135-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9a165f7-116b-4c15-9836-349bebeab135-tmp-dir\") pod \"d9a165f7-116b-4c15-9836-349bebeab135\" (UID: \"d9a165f7-116b-4c15-9836-349bebeab135\") "
Oct 31 23:25:04.617594 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.617334    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9a165f7-116b-4c15-9836-349bebeab135-home\" (UniqueName: \"kubernetes.io/empty-dir/d9a165f7-116b-4c15-9836-349bebeab135-home\") pod \"d9a165f7-116b-4c15-9836-349bebeab135\" (UID: \"d9a165f7-116b-4c15-9836-349bebeab135\") "
Oct 31 23:25:04.617594 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.617352    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9a165f7-116b-4c15-9836-349bebeab135-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9a165f7-116b-4c15-9836-349bebeab135-model-cache\") pod \"d9a165f7-116b-4c15-9836-349bebeab135\" (UID: \"d9a165f7-116b-4c15-9836-349bebeab135\") "
Oct 31 23:25:04.617594 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.617404    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1486aeca-b15c-4611-beda-0baf2e503a89-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1486aeca-b15c-4611-beda-0baf2e503a89-dshm\") pod \"1486aeca-b15c-4611-beda-0baf2e503a89\" (UID: \"1486aeca-b15c-4611-beda-0baf2e503a89\") "
Oct 31 23:25:04.617594 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.617430    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1486aeca-b15c-4611-beda-0baf2e503a89-tls-certs\" (UniqueName: \"kubernetes.io/secret/1486aeca-b15c-4611-beda-0baf2e503a89-tls-certs\") pod \"1486aeca-b15c-4611-beda-0baf2e503a89\" (UID: \"1486aeca-b15c-4611-beda-0baf2e503a89\") "
Oct 31 23:25:04.617594 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.617458    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1486aeca-b15c-4611-beda-0baf2e503a89-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1486aeca-b15c-4611-beda-0baf2e503a89-model-cache\") pod \"1486aeca-b15c-4611-beda-0baf2e503a89\" (UID: \"1486aeca-b15c-4611-beda-0baf2e503a89\") "
Oct 31 23:25:04.617594 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.617502    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1486aeca-b15c-4611-beda-0baf2e503a89-tmp-dir" pod "1486aeca-b15c-4611-beda-0baf2e503a89" (UID: "1486aeca-b15c-4611-beda-0baf2e503a89"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:04.617594 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.617564    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9a165f7-116b-4c15-9836-349bebeab135-kube-api-access-wvhks\" (UniqueName: \"kubernetes.io/projected/d9a165f7-116b-4c15-9836-349bebeab135-kube-api-access-wvhks\") pod \"d9a165f7-116b-4c15-9836-349bebeab135\" (UID: \"d9a165f7-116b-4c15-9836-349bebeab135\") "
Oct 31 23:25:04.617979 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.617616    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9a165f7-116b-4c15-9836-349bebeab135-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9a165f7-116b-4c15-9836-349bebeab135-dshm\") pod \"d9a165f7-116b-4c15-9836-349bebeab135\" (UID: \"d9a165f7-116b-4c15-9836-349bebeab135\") "
Oct 31 23:25:04.617979 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.617634    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a165f7-116b-4c15-9836-349bebeab135-home" pod "d9a165f7-116b-4c15-9836-349bebeab135" (UID: "d9a165f7-116b-4c15-9836-349bebeab135"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:04.617979 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.617647    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a165f7-116b-4c15-9836-349bebeab135-tmp-dir" pod "d9a165f7-116b-4c15-9836-349bebeab135" (UID: "d9a165f7-116b-4c15-9836-349bebeab135"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:04.617979 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.617687    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1486aeca-b15c-4611-beda-0baf2e503a89-kube-api-access-vz2lh\" (UniqueName: \"kubernetes.io/projected/1486aeca-b15c-4611-beda-0baf2e503a89-kube-api-access-vz2lh\") pod \"1486aeca-b15c-4611-beda-0baf2e503a89\" (UID: \"1486aeca-b15c-4611-beda-0baf2e503a89\") "
Oct 31 23:25:04.617979 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.617738    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1486aeca-b15c-4611-beda-0baf2e503a89-home\" (UniqueName: \"kubernetes.io/empty-dir/1486aeca-b15c-4611-beda-0baf2e503a89-home\") pod \"1486aeca-b15c-4611-beda-0baf2e503a89\" (UID: \"1486aeca-b15c-4611-beda-0baf2e503a89\") "
Oct 31 23:25:04.617979 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.617758    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1486aeca-b15c-4611-beda-0baf2e503a89-model-cache" pod "1486aeca-b15c-4611-beda-0baf2e503a89" (UID: "1486aeca-b15c-4611-beda-0baf2e503a89"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:04.617979 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.617920    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a165f7-116b-4c15-9836-349bebeab135-model-cache" pod "d9a165f7-116b-4c15-9836-349bebeab135" (UID: "d9a165f7-116b-4c15-9836-349bebeab135"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:04.618371 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.618119    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1486aeca-b15c-4611-beda-0baf2e503a89-home" pod "1486aeca-b15c-4611-beda-0baf2e503a89" (UID: "1486aeca-b15c-4611-beda-0baf2e503a89"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:04.618371 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.618306    4301 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9a165f7-116b-4c15-9836-349bebeab135-home\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:04.618371 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.618328    4301 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9a165f7-116b-4c15-9836-349bebeab135-model-cache\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:04.618371 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.618346    4301 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1486aeca-b15c-4611-beda-0baf2e503a89-model-cache\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:04.618678 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.618378    4301 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1486aeca-b15c-4611-beda-0baf2e503a89-home\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:04.618678 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.618392    4301 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1486aeca-b15c-4611-beda-0baf2e503a89-tmp-dir\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:04.618678 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.618404    4301 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9a165f7-116b-4c15-9836-349bebeab135-tmp-dir\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:04.619804 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.619777    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9a165f7-116b-4c15-9836-349bebeab135-tls-certs" pod "d9a165f7-116b-4c15-9836-349bebeab135" (UID: "d9a165f7-116b-4c15-9836-349bebeab135"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:25:04.619929 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.619833    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a165f7-116b-4c15-9836-349bebeab135-kube-api-access-wvhks" pod "d9a165f7-116b-4c15-9836-349bebeab135" (UID: "d9a165f7-116b-4c15-9836-349bebeab135"). InnerVolumeSpecName "kube-api-access-wvhks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:25:04.619929 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.619915    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1486aeca-b15c-4611-beda-0baf2e503a89-dshm" pod "1486aeca-b15c-4611-beda-0baf2e503a89" (UID: "1486aeca-b15c-4611-beda-0baf2e503a89"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:04.620129 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.620108    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a165f7-116b-4c15-9836-349bebeab135-dshm" pod "d9a165f7-116b-4c15-9836-349bebeab135" (UID: "d9a165f7-116b-4c15-9836-349bebeab135"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:04.620253 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.620238    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1486aeca-b15c-4611-beda-0baf2e503a89-kube-api-access-vz2lh" pod "1486aeca-b15c-4611-beda-0baf2e503a89" (UID: "1486aeca-b15c-4611-beda-0baf2e503a89"). InnerVolumeSpecName "kube-api-access-vz2lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:25:04.620341 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.620327    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1486aeca-b15c-4611-beda-0baf2e503a89-tls-certs" pod "1486aeca-b15c-4611-beda-0baf2e503a89" (UID: "1486aeca-b15c-4611-beda-0baf2e503a89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:25:04.709789 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.709757    4301 generic.go:356] "Generic (PLEG): container finished" podID="d9a165f7-116b-4c15-9836-349bebeab135" containerID="04ed85ed4e4daae0bc0bd6980041d7768a9d0d8b4a00a987e38724cc694e38fb" exitCode=0
Oct 31 23:25:04.710251 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.709803    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"
Oct 31 23:25:04.710251 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.709843    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" event={"ID":"d9a165f7-116b-4c15-9836-349bebeab135","Type":"ContainerDied","Data":"04ed85ed4e4daae0bc0bd6980041d7768a9d0d8b4a00a987e38724cc694e38fb"}
Oct 31 23:25:04.710251 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.709893    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc" event={"ID":"d9a165f7-116b-4c15-9836-349bebeab135","Type":"ContainerDied","Data":"c2ab3e6b2fabcd22b0a0d057ca767480fe53eafbc29201a53242351a2809a0e0"}
Oct 31 23:25:04.710251 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.709916    4301 scope.go:122] "RemoveContainer" containerID="04ed85ed4e4daae0bc0bd6980041d7768a9d0d8b4a00a987e38724cc694e38fb"
Oct 31 23:25:04.711427 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.711400    4301 generic.go:356] "Generic (PLEG): container finished" podID="1486aeca-b15c-4611-beda-0baf2e503a89" containerID="d13f078c8c0a8d358fc427ef0605270c01dc7a5937327089a1f781d81d4645b5" exitCode=0
Oct 31 23:25:04.711427 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.711423    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"
Oct 31 23:25:04.711603 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.711472    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" event={"ID":"1486aeca-b15c-4611-beda-0baf2e503a89","Type":"ContainerDied","Data":"d13f078c8c0a8d358fc427ef0605270c01dc7a5937327089a1f781d81d4645b5"}
Oct 31 23:25:04.711603 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.711511    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t" event={"ID":"1486aeca-b15c-4611-beda-0baf2e503a89","Type":"ContainerDied","Data":"c779680878376688d2c41159305275e30013e2f1cdfc84f589f90814191df7f3"}
Oct 31 23:25:04.720687 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.720663    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvhks\" (UniqueName: \"kubernetes.io/projected/d9a165f7-116b-4c15-9836-349bebeab135-kube-api-access-wvhks\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:04.720687 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.720687    4301 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9a165f7-116b-4c15-9836-349bebeab135-dshm\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:04.720885 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.720697    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz2lh\" (UniqueName: \"kubernetes.io/projected/1486aeca-b15c-4611-beda-0baf2e503a89-kube-api-access-vz2lh\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:04.720885 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.720706    4301 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9a165f7-116b-4c15-9836-349bebeab135-tls-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:04.720885 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.720716    4301 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1486aeca-b15c-4611-beda-0baf2e503a89-dshm\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:04.720885 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.720724    4301 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1486aeca-b15c-4611-beda-0baf2e503a89-tls-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:04.726792 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.726770    4301 scope.go:122] "RemoveContainer" containerID="04ed85ed4e4daae0bc0bd6980041d7768a9d0d8b4a00a987e38724cc694e38fb"
Oct 31 23:25:04.727432 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:25:04.727248    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04ed85ed4e4daae0bc0bd6980041d7768a9d0d8b4a00a987e38724cc694e38fb\": container with ID starting with 04ed85ed4e4daae0bc0bd6980041d7768a9d0d8b4a00a987e38724cc694e38fb not found: ID does not exist" containerID="04ed85ed4e4daae0bc0bd6980041d7768a9d0d8b4a00a987e38724cc694e38fb"
Oct 31 23:25:04.727432 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.727286    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04ed85ed4e4daae0bc0bd6980041d7768a9d0d8b4a00a987e38724cc694e38fb"} err="failed to get container status \"04ed85ed4e4daae0bc0bd6980041d7768a9d0d8b4a00a987e38724cc694e38fb\": rpc error: code = NotFound desc = could not find container \"04ed85ed4e4daae0bc0bd6980041d7768a9d0d8b4a00a987e38724cc694e38fb\": container with ID starting with 04ed85ed4e4daae0bc0bd6980041d7768a9d0d8b4a00a987e38724cc694e38fb not found: ID does not exist"
Oct 31 23:25:04.727432 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.727308    4301 scope.go:122] "RemoveContainer" containerID="d13f078c8c0a8d358fc427ef0605270c01dc7a5937327089a1f781d81d4645b5"
Oct 31 23:25:04.739269 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.739240    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"]
Oct 31 23:25:04.739395 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.739369    4301 scope.go:122] "RemoveContainer" containerID="d13f078c8c0a8d358fc427ef0605270c01dc7a5937327089a1f781d81d4645b5"
Oct 31 23:25:04.739717 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:25:04.739694    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d13f078c8c0a8d358fc427ef0605270c01dc7a5937327089a1f781d81d4645b5\": container with ID starting with d13f078c8c0a8d358fc427ef0605270c01dc7a5937327089a1f781d81d4645b5 not found: ID does not exist" containerID="d13f078c8c0a8d358fc427ef0605270c01dc7a5937327089a1f781d81d4645b5"
Oct 31 23:25:04.739830 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.739723    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d13f078c8c0a8d358fc427ef0605270c01dc7a5937327089a1f781d81d4645b5"} err="failed to get container status \"d13f078c8c0a8d358fc427ef0605270c01dc7a5937327089a1f781d81d4645b5\": rpc error: code = NotFound desc = could not find container \"d13f078c8c0a8d358fc427ef0605270c01dc7a5937327089a1f781d81d4645b5\": container with ID starting with d13f078c8c0a8d358fc427ef0605270c01dc7a5937327089a1f781d81d4645b5 not found: ID does not exist"
Oct 31 23:25:04.746159 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.746132    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-67fce9/simulator-w3-67fce9-kserve-9c6ddd964-46q2t"]
Oct 31 23:25:04.757845 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.757817    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"]
Oct 31 23:25:04.763129 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.763104    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-67fce9/distinct-2-w3-67fce9-kserve-dd6fd597f-7rqtc"]
Oct 31 23:25:04.934204 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:04.934174    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f98a428-bzspr"
Oct 31 23:25:05.033709 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:05.033677    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fbc8cf6-9cce-414c-9146-8a2ebd840fe0-kube-api-access-pflfk\" (UniqueName: \"kubernetes.io/projected/2fbc8cf6-9cce-414c-9146-8a2ebd840fe0-kube-api-access-pflfk\") pod \"2fbc8cf6-9cce-414c-9146-8a2ebd840fe0\" (UID: \"2fbc8cf6-9cce-414c-9146-8a2ebd840fe0\") "
Oct 31 23:25:05.033897 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:05.033795    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fbc8cf6-9cce-414c-9146-8a2ebd840fe0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fbc8cf6-9cce-414c-9146-8a2ebd840fe0-maas-api-openshift-ca\") pod \"2fbc8cf6-9cce-414c-9146-8a2ebd840fe0\" (UID: \"2fbc8cf6-9cce-414c-9146-8a2ebd840fe0\") "
Oct 31 23:25:05.033897 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:05.033859    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fbc8cf6-9cce-414c-9146-8a2ebd840fe0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fbc8cf6-9cce-414c-9146-8a2ebd840fe0-maas-api-xks-ca\") pod \"2fbc8cf6-9cce-414c-9146-8a2ebd840fe0\" (UID: \"2fbc8cf6-9cce-414c-9146-8a2ebd840fe0\") "
Oct 31 23:25:05.034924 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:05.034903    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fbc8cf6-9cce-414c-9146-8a2ebd840fe0-maas-api-openshift-ca" pod "2fbc8cf6-9cce-414c-9146-8a2ebd840fe0" (UID: "2fbc8cf6-9cce-414c-9146-8a2ebd840fe0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:25:05.035009 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:25:05.034961    4301 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3bc4bb3_e967_4fe3_a6b9_71377f8f6047.slice/crio-conmon-5de4cae8fa83ea1bedeac765e1e1ae9941651750c3419eadbef02ad843453b20.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 23:25:05.037007 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:05.036962    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fbc8cf6-9cce-414c-9146-8a2ebd840fe0-maas-api-xks-ca" pod "2fbc8cf6-9cce-414c-9146-8a2ebd840fe0" (UID: "2fbc8cf6-9cce-414c-9146-8a2ebd840fe0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:25:05.037110 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:05.037025    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fbc8cf6-9cce-414c-9146-8a2ebd840fe0-kube-api-access-pflfk" pod "2fbc8cf6-9cce-414c-9146-8a2ebd840fe0" (UID: "2fbc8cf6-9cce-414c-9146-8a2ebd840fe0"). InnerVolumeSpecName "kube-api-access-pflfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:25:05.135501 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:05.135464    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fbc8cf6-9cce-414c-9146-8a2ebd840fe0-maas-api-xks-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:05.135734 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:05.135714    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pflfk\" (UniqueName: \"kubernetes.io/projected/2fbc8cf6-9cce-414c-9146-8a2ebd840fe0-kube-api-access-pflfk\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:05.135840 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:05.135828    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fbc8cf6-9cce-414c-9146-8a2ebd840fe0-maas-api-openshift-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:05.355277 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:05.354925    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d7eed-7c97f4ff9f-whnrt"
Oct 31 23:25:05.416611 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:05.414244    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" path="/var/lib/kubelet/pods/1486aeca-b15c-4611-beda-0baf2e503a89/volumes"
Oct 31 23:25:05.417803 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:05.417778    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9a165f7-116b-4c15-9836-349bebeab135" path="/var/lib/kubelet/pods/d9a165f7-116b-4c15-9836-349bebeab135/volumes"
Oct 31 23:25:05.452687 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:05.452661    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3bc4bb3-e967-4fe3-a6b9-71377f8f6047-kube-api-access-zq6g6\" (UniqueName: \"kubernetes.io/projected/a3bc4bb3-e967-4fe3-a6b9-71377f8f6047-kube-api-access-zq6g6\") pod \"a3bc4bb3-e967-4fe3-a6b9-71377f8f6047\" (UID: \"a3bc4bb3-e967-4fe3-a6b9-71377f8f6047\") "
Oct 31 23:25:05.452836 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:05.452830    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3bc4bb3-e967-4fe3-a6b9-71377f8f6047-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3bc4bb3-e967-4fe3-a6b9-71377f8f6047-plugins-config-volume\") pod \"a3bc4bb3-e967-4fe3-a6b9-71377f8f6047\" (UID: \"a3bc4bb3-e967-4fe3-a6b9-71377f8f6047\") "
Oct 31 23:25:05.453198 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:05.453167    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3bc4bb3-e967-4fe3-a6b9-71377f8f6047-plugins-config-volume" pod "a3bc4bb3-e967-4fe3-a6b9-71377f8f6047" (UID: "a3bc4bb3-e967-4fe3-a6b9-71377f8f6047"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:25:05.453306 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:05.453274    4301 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3bc4bb3-e967-4fe3-a6b9-71377f8f6047-plugins-config-volume\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:05.454845 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:05.454822    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3bc4bb3-e967-4fe3-a6b9-71377f8f6047-kube-api-access-zq6g6" pod "a3bc4bb3-e967-4fe3-a6b9-71377f8f6047" (UID: "a3bc4bb3-e967-4fe3-a6b9-71377f8f6047"). InnerVolumeSpecName "kube-api-access-zq6g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:25:05.554393 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:05.554293    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq6g6\" (UniqueName: \"kubernetes.io/projected/a3bc4bb3-e967-4fe3-a6b9-71377f8f6047-kube-api-access-zq6g6\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:05.724638 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:05.724609    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f98a428-bzspr"
Oct 31 23:25:05.725160 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:05.724608    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f98a428-bzspr" event={"ID":"2fbc8cf6-9cce-414c-9146-8a2ebd840fe0","Type":"ContainerDied","Data":"73ae308343dc093ded53301d85c24f740de9e5662604640d6283fc3e83029d2c"}
Oct 31 23:25:05.725160 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:05.724785    4301 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73ae308343dc093ded53301d85c24f740de9e5662604640d6283fc3e83029d2c"
Oct 31 23:25:05.726474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:05.726445    4301 generic.go:356] "Generic (PLEG): container finished" podID="a3bc4bb3-e967-4fe3-a6b9-71377f8f6047" containerID="5de4cae8fa83ea1bedeac765e1e1ae9941651750c3419eadbef02ad843453b20" exitCode=0
Oct 31 23:25:05.726617 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:05.726487    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d7eed-7c97f4ff9f-whnrt" event={"ID":"a3bc4bb3-e967-4fe3-a6b9-71377f8f6047","Type":"ContainerDied","Data":"5de4cae8fa83ea1bedeac765e1e1ae9941651750c3419eadbef02ad843453b20"}
Oct 31 23:25:05.726617 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:05.726521    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d7eed-7c97f4ff9f-whnrt"
Oct 31 23:25:05.726617 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:05.726538    4301 scope.go:122] "RemoveContainer" containerID="5de4cae8fa83ea1bedeac765e1e1ae9941651750c3419eadbef02ad843453b20"
Oct 31 23:25:05.726855 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:05.726526    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d7eed-7c97f4ff9f-whnrt" event={"ID":"a3bc4bb3-e967-4fe3-a6b9-71377f8f6047","Type":"ContainerDied","Data":"a61a1b936e354a3bbfb65ad2d30566b13b125aa90e9bb4233c025fd540d748f5"}
Oct 31 23:25:05.741460 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:05.741435    4301 scope.go:122] "RemoveContainer" containerID="5de4cae8fa83ea1bedeac765e1e1ae9941651750c3419eadbef02ad843453b20"
Oct 31 23:25:05.741778 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:25:05.741752    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5de4cae8fa83ea1bedeac765e1e1ae9941651750c3419eadbef02ad843453b20\": container with ID starting with 5de4cae8fa83ea1bedeac765e1e1ae9941651750c3419eadbef02ad843453b20 not found: ID does not exist" containerID="5de4cae8fa83ea1bedeac765e1e1ae9941651750c3419eadbef02ad843453b20"
Oct 31 23:25:05.741960 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:05.741927    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5de4cae8fa83ea1bedeac765e1e1ae9941651750c3419eadbef02ad843453b20"} err="failed to get container status \"5de4cae8fa83ea1bedeac765e1e1ae9941651750c3419eadbef02ad843453b20\": rpc error: code = NotFound desc = could not find container \"5de4cae8fa83ea1bedeac765e1e1ae9941651750c3419eadbef02ad843453b20\": container with ID starting with 5de4cae8fa83ea1bedeac765e1e1ae9941651750c3419eadbef02ad843453b20 not found: ID does not exist"
Oct 31 23:25:05.756005 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:05.755978    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7d7eed-7c97f4ff9f-whnrt"]
Oct 31 23:25:05.760725 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:05.760701    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7d7eed-7c97f4ff9f-whnrt"]
Oct 31 23:25:05.904708 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:05.904672    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:25:07.403656 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:07.403623    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3bc4bb3-e967-4fe3-a6b9-71377f8f6047" path="/var/lib/kubelet/pods/a3bc4bb3-e967-4fe3-a6b9-71377f8f6047/volumes"
Oct 31 23:25:07.570320 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:07.570290    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"]
Oct 31 23:25:07.570544 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:07.570516    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t" podUID="fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5" containerName="istio-proxy" containerID="cri-o://5494c25f1a03d3cbc311c4194963bf23987c9df812cb86599e8fde7524ff4f2d" gracePeriod=30
Oct 31 23:25:07.573776 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:07.573751    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"]
Oct 31 23:25:07.573956 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:07.573937    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9" podUID="f7b097bb-1a53-48cc-8499-f13014278d10" containerName="istio-proxy" containerID="cri-o://0b1a39944e5d8230624e6db233fd50248f84730c3c5b5c9ded424d0a432d221f" gracePeriod=30
Oct 31 23:25:07.581065 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:07.581042    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"]
Oct 31 23:25:07.581292 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:07.581268    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw" podUID="2abed568-6cc8-4336-b427-b8a976686c3f" containerName="istio-proxy" containerID="cri-o://23ab4029c0559ed24c259d479471c886a4ee16cfe31e79979cf99e43f8cf7bdc" gracePeriod=30
Oct 31 23:25:09.244254 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:09.244172    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"]
Oct 31 23:25:09.260667 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:09.260014    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2" podUID="bf839f5e-2570-44bd-9c62-7a65ea1590ed" containerName="istio-proxy" containerID="cri-o://4706b538ebf255543d9badb25c61d0a5dc8efc72762b1e2dd71a58e52bc31c1e" gracePeriod=30
Oct 31 23:25:09.264926 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:09.264886    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"]
Oct 31 23:25:09.265244 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:09.265221    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg" podUID="d0195c12-01bd-4f4a-a72a-b608150516f2" containerName="istio-proxy" containerID="cri-o://b55f6e37daaf160cc50371071ce1abe701dcc03711497ddb0828c5bcba7adb06" gracePeriod=30
Oct 31 23:25:09.752283 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:09.752246    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr"]
Oct 31 23:25:09.752577 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:09.752549    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr" podUID="9aac5700-d634-407f-98f1-e7ab6beeebc8" containerName="maas-api" containerID="cri-o://70018d80bdc77c29ee2bae65e41b87b7300760a142c944a0c6f74f96c1f5b1c6" gracePeriod=30
Oct 31 23:25:10.148657 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:10.148632    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr"
Oct 31 23:25:10.211948 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:10.211913    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aac5700-d634-407f-98f1-e7ab6beeebc8-kube-api-access-6kzkr\" (UniqueName: \"kubernetes.io/projected/9aac5700-d634-407f-98f1-e7ab6beeebc8-kube-api-access-6kzkr\") pod \"9aac5700-d634-407f-98f1-e7ab6beeebc8\" (UID: \"9aac5700-d634-407f-98f1-e7ab6beeebc8\") "
Oct 31 23:25:10.212116 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:10.212011    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9aac5700-d634-407f-98f1-e7ab6beeebc8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aac5700-d634-407f-98f1-e7ab6beeebc8-openshift-service-ca\") pod \"9aac5700-d634-407f-98f1-e7ab6beeebc8\" (UID: \"9aac5700-d634-407f-98f1-e7ab6beeebc8\") "
Oct 31 23:25:10.212165 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:10.212136    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9aac5700-d634-407f-98f1-e7ab6beeebc8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aac5700-d634-407f-98f1-e7ab6beeebc8-maas-api-tls\") pod \"9aac5700-d634-407f-98f1-e7ab6beeebc8\" (UID: \"9aac5700-d634-407f-98f1-e7ab6beeebc8\") "
Oct 31 23:25:10.212165 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:10.212154    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9aac5700-d634-407f-98f1-e7ab6beeebc8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aac5700-d634-407f-98f1-e7ab6beeebc8-metrics-certs\") pod \"9aac5700-d634-407f-98f1-e7ab6beeebc8\" (UID: \"9aac5700-d634-407f-98f1-e7ab6beeebc8\") "
Oct 31 23:25:10.212463 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:10.212432    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aac5700-d634-407f-98f1-e7ab6beeebc8-openshift-service-ca" pod "9aac5700-d634-407f-98f1-e7ab6beeebc8" (UID: "9aac5700-d634-407f-98f1-e7ab6beeebc8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:25:10.212616 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:10.212597    4301 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aac5700-d634-407f-98f1-e7ab6beeebc8-openshift-service-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:10.214328 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:10.214302    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aac5700-d634-407f-98f1-e7ab6beeebc8-kube-api-access-6kzkr" pod "9aac5700-d634-407f-98f1-e7ab6beeebc8" (UID: "9aac5700-d634-407f-98f1-e7ab6beeebc8"). InnerVolumeSpecName "kube-api-access-6kzkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:25:10.214449 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:10.214336    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aac5700-d634-407f-98f1-e7ab6beeebc8-metrics-certs" pod "9aac5700-d634-407f-98f1-e7ab6beeebc8" (UID: "9aac5700-d634-407f-98f1-e7ab6beeebc8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:25:10.214508 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:10.214489    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aac5700-d634-407f-98f1-e7ab6beeebc8-maas-api-tls" pod "9aac5700-d634-407f-98f1-e7ab6beeebc8" (UID: "9aac5700-d634-407f-98f1-e7ab6beeebc8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:25:10.318416 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:10.318352    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kzkr\" (UniqueName: \"kubernetes.io/projected/9aac5700-d634-407f-98f1-e7ab6beeebc8-kube-api-access-6kzkr\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:10.318416 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:10.318416    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aac5700-d634-407f-98f1-e7ab6beeebc8-maas-api-tls\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:10.318966 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:10.318432    4301 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aac5700-d634-407f-98f1-e7ab6beeebc8-metrics-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:10.845190 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:10.845159    4301 generic.go:356] "Generic (PLEG): container finished" podID="9aac5700-d634-407f-98f1-e7ab6beeebc8" containerID="70018d80bdc77c29ee2bae65e41b87b7300760a142c944a0c6f74f96c1f5b1c6" exitCode=0
Oct 31 23:25:10.845536 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:10.845210    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr"
Oct 31 23:25:10.845536 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:10.845248    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr" event={"ID":"9aac5700-d634-407f-98f1-e7ab6beeebc8","Type":"ContainerDied","Data":"70018d80bdc77c29ee2bae65e41b87b7300760a142c944a0c6f74f96c1f5b1c6"}
Oct 31 23:25:10.845536 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:10.845314    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr" event={"ID":"9aac5700-d634-407f-98f1-e7ab6beeebc8","Type":"ContainerDied","Data":"d2e65365cd42777a657401a164b02f846fafb9f6d94582ce3ad012be0e8bb9bd"}
Oct 31 23:25:10.845536 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:10.845345    4301 scope.go:122] "RemoveContainer" containerID="70018d80bdc77c29ee2bae65e41b87b7300760a142c944a0c6f74f96c1f5b1c6"
Oct 31 23:25:10.855863 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:10.855846    4301 scope.go:122] "RemoveContainer" containerID="70018d80bdc77c29ee2bae65e41b87b7300760a142c944a0c6f74f96c1f5b1c6"
Oct 31 23:25:10.856174 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:25:10.856155    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70018d80bdc77c29ee2bae65e41b87b7300760a142c944a0c6f74f96c1f5b1c6\": container with ID starting with 70018d80bdc77c29ee2bae65e41b87b7300760a142c944a0c6f74f96c1f5b1c6 not found: ID does not exist" containerID="70018d80bdc77c29ee2bae65e41b87b7300760a142c944a0c6f74f96c1f5b1c6"
Oct 31 23:25:10.856230 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:10.856183    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70018d80bdc77c29ee2bae65e41b87b7300760a142c944a0c6f74f96c1f5b1c6"} err="failed to get container status \"70018d80bdc77c29ee2bae65e41b87b7300760a142c944a0c6f74f96c1f5b1c6\": rpc error: code = NotFound desc = could not find container \"70018d80bdc77c29ee2bae65e41b87b7300760a142c944a0c6f74f96c1f5b1c6\": container with ID starting with 70018d80bdc77c29ee2bae65e41b87b7300760a142c944a0c6f74f96c1f5b1c6 not found: ID does not exist"
Oct 31 23:25:10.867637 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:10.867612    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr"]
Oct 31 23:25:10.871538 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:10.871516    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-67fce9-b6f4695d4-7t2rr"]
Oct 31 23:25:11.402826 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:11.402794    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aac5700-d634-407f-98f1-e7ab6beeebc8" path="/var/lib/kubelet/pods/9aac5700-d634-407f-98f1-e7ab6beeebc8/volumes"
Oct 31 23:25:11.405816 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:11.405787    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" containerID="cri-o://8fb839d012c9f4a9840c24ad3e5fce6c480abe9ead065bcefa09a2322af595e3" gracePeriod=45
Oct 31 23:25:11.630371 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:11.630321    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": dial tcp 10.133.0.48:8000: connect: connection refused"
Oct 31 23:25:11.751446 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:11.751415    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:25:11.843400 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:11.843370    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570b97e3-1e23-474c-beb7-9438692c5508-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/570b97e3-1e23-474c-beb7-9438692c5508-tmp-dir\") pod \"570b97e3-1e23-474c-beb7-9438692c5508\" (UID: \"570b97e3-1e23-474c-beb7-9438692c5508\") "
Oct 31 23:25:11.843563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:11.843442    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570b97e3-1e23-474c-beb7-9438692c5508-home\" (UniqueName: \"kubernetes.io/empty-dir/570b97e3-1e23-474c-beb7-9438692c5508-home\") pod \"570b97e3-1e23-474c-beb7-9438692c5508\" (UID: \"570b97e3-1e23-474c-beb7-9438692c5508\") "
Oct 31 23:25:11.843563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:11.843493    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570b97e3-1e23-474c-beb7-9438692c5508-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/570b97e3-1e23-474c-beb7-9438692c5508-model-cache\") pod \"570b97e3-1e23-474c-beb7-9438692c5508\" (UID: \"570b97e3-1e23-474c-beb7-9438692c5508\") "
Oct 31 23:25:11.843688 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:11.843572    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/570b97e3-1e23-474c-beb7-9438692c5508-kube-api-access-5c95b\" (UniqueName: \"kubernetes.io/projected/570b97e3-1e23-474c-beb7-9438692c5508-kube-api-access-5c95b\") pod \"570b97e3-1e23-474c-beb7-9438692c5508\" (UID: \"570b97e3-1e23-474c-beb7-9438692c5508\") "
Oct 31 23:25:11.843688 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:11.843610    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570b97e3-1e23-474c-beb7-9438692c5508-dshm\" (UniqueName: \"kubernetes.io/empty-dir/570b97e3-1e23-474c-beb7-9438692c5508-dshm\") pod \"570b97e3-1e23-474c-beb7-9438692c5508\" (UID: \"570b97e3-1e23-474c-beb7-9438692c5508\") "
Oct 31 23:25:11.843688 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:11.843630    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/570b97e3-1e23-474c-beb7-9438692c5508-tls-certs\" (UniqueName: \"kubernetes.io/secret/570b97e3-1e23-474c-beb7-9438692c5508-tls-certs\") pod \"570b97e3-1e23-474c-beb7-9438692c5508\" (UID: \"570b97e3-1e23-474c-beb7-9438692c5508\") "
Oct 31 23:25:11.843688 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:11.843653    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570b97e3-1e23-474c-beb7-9438692c5508-tmp-dir" pod "570b97e3-1e23-474c-beb7-9438692c5508" (UID: "570b97e3-1e23-474c-beb7-9438692c5508"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:11.843894 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:11.843710    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570b97e3-1e23-474c-beb7-9438692c5508-home" pod "570b97e3-1e23-474c-beb7-9438692c5508" (UID: "570b97e3-1e23-474c-beb7-9438692c5508"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:11.843894 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:11.843790    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570b97e3-1e23-474c-beb7-9438692c5508-model-cache" pod "570b97e3-1e23-474c-beb7-9438692c5508" (UID: "570b97e3-1e23-474c-beb7-9438692c5508"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:11.844078 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:11.844059    4301 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/570b97e3-1e23-474c-beb7-9438692c5508-home\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:11.844136 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:11.844083    4301 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/570b97e3-1e23-474c-beb7-9438692c5508-model-cache\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:11.844136 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:11.844097    4301 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/570b97e3-1e23-474c-beb7-9438692c5508-tmp-dir\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:11.846044 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:11.846021    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/570b97e3-1e23-474c-beb7-9438692c5508-tls-certs" pod "570b97e3-1e23-474c-beb7-9438692c5508" (UID: "570b97e3-1e23-474c-beb7-9438692c5508"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:25:11.846044 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:11.846034    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570b97e3-1e23-474c-beb7-9438692c5508-dshm" pod "570b97e3-1e23-474c-beb7-9438692c5508" (UID: "570b97e3-1e23-474c-beb7-9438692c5508"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:11.846222 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:11.846065    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/570b97e3-1e23-474c-beb7-9438692c5508-kube-api-access-5c95b" pod "570b97e3-1e23-474c-beb7-9438692c5508" (UID: "570b97e3-1e23-474c-beb7-9438692c5508"). InnerVolumeSpecName "kube-api-access-5c95b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:25:11.855731 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:11.855708    4301 generic.go:356] "Generic (PLEG): container finished" podID="570b97e3-1e23-474c-beb7-9438692c5508" containerID="8fb839d012c9f4a9840c24ad3e5fce6c480abe9ead065bcefa09a2322af595e3" exitCode=0
Oct 31 23:25:11.855871 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:11.855745    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" event={"ID":"570b97e3-1e23-474c-beb7-9438692c5508","Type":"ContainerDied","Data":"8fb839d012c9f4a9840c24ad3e5fce6c480abe9ead065bcefa09a2322af595e3"}
Oct 31 23:25:11.855871 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:11.855770    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd" event={"ID":"570b97e3-1e23-474c-beb7-9438692c5508","Type":"ContainerDied","Data":"81768cb8b32f79b624805fdaf684efbb902b9436ae9568118f0095c1bb4654bc"}
Oct 31 23:25:11.855871 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:11.855771    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"
Oct 31 23:25:11.855871 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:11.855785    4301 scope.go:122] "RemoveContainer" containerID="8fb839d012c9f4a9840c24ad3e5fce6c480abe9ead065bcefa09a2322af595e3"
Oct 31 23:25:11.867987 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:11.867970    4301 scope.go:122] "RemoveContainer" containerID="8fb839d012c9f4a9840c24ad3e5fce6c480abe9ead065bcefa09a2322af595e3"
Oct 31 23:25:11.868257 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:25:11.868237    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fb839d012c9f4a9840c24ad3e5fce6c480abe9ead065bcefa09a2322af595e3\": container with ID starting with 8fb839d012c9f4a9840c24ad3e5fce6c480abe9ead065bcefa09a2322af595e3 not found: ID does not exist" containerID="8fb839d012c9f4a9840c24ad3e5fce6c480abe9ead065bcefa09a2322af595e3"
Oct 31 23:25:11.868310 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:11.868264    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb839d012c9f4a9840c24ad3e5fce6c480abe9ead065bcefa09a2322af595e3"} err="failed to get container status \"8fb839d012c9f4a9840c24ad3e5fce6c480abe9ead065bcefa09a2322af595e3\": rpc error: code = NotFound desc = could not find container \"8fb839d012c9f4a9840c24ad3e5fce6c480abe9ead065bcefa09a2322af595e3\": container with ID starting with 8fb839d012c9f4a9840c24ad3e5fce6c480abe9ead065bcefa09a2322af595e3 not found: ID does not exist"
Oct 31 23:25:11.884602 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:11.884561    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"]
Oct 31 23:25:11.886928 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:11.886906    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-04c5c2/distinct-w1-04c5c2-kserve-79fc6d9c89-shxwd"]
Oct 31 23:25:11.945473 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:11.945444    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c95b\" (UniqueName: \"kubernetes.io/projected/570b97e3-1e23-474c-beb7-9438692c5508-kube-api-access-5c95b\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:11.945473 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:11.945467    4301 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/570b97e3-1e23-474c-beb7-9438692c5508-dshm\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:11.945473 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:11.945476    4301 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/570b97e3-1e23-474c-beb7-9438692c5508-tls-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:12.865857 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:12.865770    4301 generic.go:356] "Generic (PLEG): container finished" podID="2abed568-6cc8-4336-b427-b8a976686c3f" containerID="23ab4029c0559ed24c259d479471c886a4ee16cfe31e79979cf99e43f8cf7bdc" exitCode=0
Oct 31 23:25:12.866263 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:12.865943    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw" event={"ID":"2abed568-6cc8-4336-b427-b8a976686c3f","Type":"ContainerDied","Data":"23ab4029c0559ed24c259d479471c886a4ee16cfe31e79979cf99e43f8cf7bdc"}
Oct 31 23:25:12.867910 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:12.867885    4301 generic.go:356] "Generic (PLEG): container finished" podID="fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5" containerID="5494c25f1a03d3cbc311c4194963bf23987c9df812cb86599e8fde7524ff4f2d" exitCode=0
Oct 31 23:25:12.868039 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:12.867958    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t" event={"ID":"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5","Type":"ContainerDied","Data":"5494c25f1a03d3cbc311c4194963bf23987c9df812cb86599e8fde7524ff4f2d"}
Oct 31 23:25:12.871305 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:12.871160    4301 generic.go:356] "Generic (PLEG): container finished" podID="f7b097bb-1a53-48cc-8499-f13014278d10" containerID="0b1a39944e5d8230624e6db233fd50248f84730c3c5b5c9ded424d0a432d221f" exitCode=0
Oct 31 23:25:12.871305 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:12.871218    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9" event={"ID":"f7b097bb-1a53-48cc-8499-f13014278d10","Type":"ContainerDied","Data":"0b1a39944e5d8230624e6db233fd50248f84730c3c5b5c9ded424d0a432d221f"}
Oct 31 23:25:12.894783 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:12.894759    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:25:12.944623 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:12.944596    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:25:12.995860 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:12.995830    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:25:13.062762 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.062727    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istio-data\") pod \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") "
Oct 31 23:25:13.062955 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.062767    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-workload-certs\") pod \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") "
Oct 31 23:25:13.062955 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.062794    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istio-envoy\") pod \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") "
Oct 31 23:25:13.062955 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.062820    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-kube-api-access-kmjhs\" (UniqueName: \"kubernetes.io/projected/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-kube-api-access-kmjhs\") pod \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") "
Oct 31 23:25:13.062955 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.062838    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b097bb-1a53-48cc-8499-f13014278d10-istio-token\" (UniqueName: \"kubernetes.io/projected/f7b097bb-1a53-48cc-8499-f13014278d10-istio-token\") pod \"f7b097bb-1a53-48cc-8499-f13014278d10\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") "
Oct 31 23:25:13.062955 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.062869    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istiod-ca-cert\") pod \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") "
Oct 31 23:25:13.062955 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.062900    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istio-token\" (UniqueName: \"kubernetes.io/projected/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istio-token\") pod \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") "
Oct 31 23:25:13.063300 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.063037    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-workload-socket\") pod \"f7b097bb-1a53-48cc-8499-f13014278d10\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") "
Oct 31 23:25:13.063300 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.063085    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-workload-certs" pod "fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5" (UID: "fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:13.063300 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.063113    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7b097bb-1a53-48cc-8499-f13014278d10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b097bb-1a53-48cc-8499-f13014278d10-istio-podinfo\") pod \"f7b097bb-1a53-48cc-8499-f13014278d10\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") "
Oct 31 23:25:13.063300 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.063188    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-istio-envoy\") pod \"f7b097bb-1a53-48cc-8499-f13014278d10\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") "
Oct 31 23:25:13.063300 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.063216    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-workload-socket\") pod \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") "
Oct 31 23:25:13.063300 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.063246    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-workload-certs\") pod \"f7b097bb-1a53-48cc-8499-f13014278d10\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") "
Oct 31 23:25:13.063625 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.063333    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-workload-socket" pod "f7b097bb-1a53-48cc-8499-f13014278d10" (UID: "f7b097bb-1a53-48cc-8499-f13014278d10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:13.063625 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.063376    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7b097bb-1a53-48cc-8499-f13014278d10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b097bb-1a53-48cc-8499-f13014278d10-istiod-ca-cert\") pod \"f7b097bb-1a53-48cc-8499-f13014278d10\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") "
Oct 31 23:25:13.063625 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.063413    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istiod-ca-cert" pod "fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5" (UID: "fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:25:13.063625 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.063425    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istio-podinfo\") pod \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") "
Oct 31 23:25:13.063625 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.063490    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-credential-socket\") pod \"f7b097bb-1a53-48cc-8499-f13014278d10\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") "
Oct 31 23:25:13.063625 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.063541    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b097bb-1a53-48cc-8499-f13014278d10-kube-api-access-6lggv\" (UniqueName: \"kubernetes.io/projected/f7b097bb-1a53-48cc-8499-f13014278d10-kube-api-access-6lggv\") pod \"f7b097bb-1a53-48cc-8499-f13014278d10\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") "
Oct 31 23:25:13.063951 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.063636    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-credential-socket\") pod \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\" (UID: \"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5\") "
Oct 31 23:25:13.063951 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.063694    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-istio-data\") pod \"f7b097bb-1a53-48cc-8499-f13014278d10\" (UID: \"f7b097bb-1a53-48cc-8499-f13014278d10\") "
Oct 31 23:25:13.064187 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.064166    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istio-data" pod "fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5" (UID: "fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:13.064542 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.064459    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:13.064542 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.064484    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:13.064542 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.064500    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:13.064542 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.064514    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:13.064879 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.064632    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-workload-certs" pod "f7b097bb-1a53-48cc-8499-f13014278d10" (UID: "f7b097bb-1a53-48cc-8499-f13014278d10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:13.064879 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.064871    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-workload-socket" pod "fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5" (UID: "fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:13.065406 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.065339    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-credential-socket" pod "f7b097bb-1a53-48cc-8499-f13014278d10" (UID: "f7b097bb-1a53-48cc-8499-f13014278d10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:13.065720 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.065699    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istio-envoy" pod "fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5" (UID: "fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:13.065838 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.065742    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istio-podinfo" pod "fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5" (UID: "fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:25:13.065838 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.065797    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-kube-api-access-kmjhs" pod "fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5" (UID: "fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5"). InnerVolumeSpecName "kube-api-access-kmjhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:25:13.065997 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.065939    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-credential-socket" pod "fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5" (UID: "fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:13.065997 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.065942    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b097bb-1a53-48cc-8499-f13014278d10-istio-token" pod "f7b097bb-1a53-48cc-8499-f13014278d10" (UID: "f7b097bb-1a53-48cc-8499-f13014278d10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:25:13.066146 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.065999    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-istio-data" pod "f7b097bb-1a53-48cc-8499-f13014278d10" (UID: "f7b097bb-1a53-48cc-8499-f13014278d10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:13.066227 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.066206    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b097bb-1a53-48cc-8499-f13014278d10-istiod-ca-cert" pod "f7b097bb-1a53-48cc-8499-f13014278d10" (UID: "f7b097bb-1a53-48cc-8499-f13014278d10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:25:13.066267 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.066249    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istio-token" pod "fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5" (UID: "fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:25:13.066800 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.066786    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-istio-envoy" pod "f7b097bb-1a53-48cc-8499-f13014278d10" (UID: "f7b097bb-1a53-48cc-8499-f13014278d10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:13.067069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.067052    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7b097bb-1a53-48cc-8499-f13014278d10-istio-podinfo" pod "f7b097bb-1a53-48cc-8499-f13014278d10" (UID: "f7b097bb-1a53-48cc-8499-f13014278d10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:25:13.067421 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.067397    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b097bb-1a53-48cc-8499-f13014278d10-kube-api-access-6lggv" pod "f7b097bb-1a53-48cc-8499-f13014278d10" (UID: "f7b097bb-1a53-48cc-8499-f13014278d10"). InnerVolumeSpecName "kube-api-access-6lggv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:25:13.164914 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.164881    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-credential-socket\") pod \"2abed568-6cc8-4336-b427-b8a976686c3f\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") "
Oct 31 23:25:13.165118 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.164921    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2abed568-6cc8-4336-b427-b8a976686c3f-kube-api-access-rvdtp\" (UniqueName: \"kubernetes.io/projected/2abed568-6cc8-4336-b427-b8a976686c3f-kube-api-access-rvdtp\") pod \"2abed568-6cc8-4336-b427-b8a976686c3f\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") "
Oct 31 23:25:13.165118 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.164971    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2abed568-6cc8-4336-b427-b8a976686c3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2abed568-6cc8-4336-b427-b8a976686c3f-istio-podinfo\") pod \"2abed568-6cc8-4336-b427-b8a976686c3f\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") "
Oct 31 23:25:13.165118 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.164996    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2abed568-6cc8-4336-b427-b8a976686c3f-istio-token\" (UniqueName: \"kubernetes.io/projected/2abed568-6cc8-4336-b427-b8a976686c3f-istio-token\") pod \"2abed568-6cc8-4336-b427-b8a976686c3f\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") "
Oct 31 23:25:13.165118 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.165060    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-workload-socket\") pod \"2abed568-6cc8-4336-b427-b8a976686c3f\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") "
Oct 31 23:25:13.165118 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.165078    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-istio-envoy\") pod \"2abed568-6cc8-4336-b427-b8a976686c3f\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") "
Oct 31 23:25:13.165445 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.165162    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-credential-socket" pod "2abed568-6cc8-4336-b427-b8a976686c3f" (UID: "2abed568-6cc8-4336-b427-b8a976686c3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:13.165445 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.165174    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-workload-certs\") pod \"2abed568-6cc8-4336-b427-b8a976686c3f\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") "
Oct 31 23:25:13.165445 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.165247    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2abed568-6cc8-4336-b427-b8a976686c3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2abed568-6cc8-4336-b427-b8a976686c3f-istiod-ca-cert\") pod \"2abed568-6cc8-4336-b427-b8a976686c3f\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") "
Oct 31 23:25:13.165445 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.165298    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-istio-data\") pod \"2abed568-6cc8-4336-b427-b8a976686c3f\" (UID: \"2abed568-6cc8-4336-b427-b8a976686c3f\") "
Oct 31 23:25:13.165645 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.165454    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-workload-certs" pod "2abed568-6cc8-4336-b427-b8a976686c3f" (UID: "2abed568-6cc8-4336-b427-b8a976686c3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:13.165761 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.165739    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-istio-data" pod "2abed568-6cc8-4336-b427-b8a976686c3f" (UID: "2abed568-6cc8-4336-b427-b8a976686c3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:13.165813 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.165761    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2abed568-6cc8-4336-b427-b8a976686c3f-istiod-ca-cert" pod "2abed568-6cc8-4336-b427-b8a976686c3f" (UID: "2abed568-6cc8-4336-b427-b8a976686c3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:25:13.165908 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.165885    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lggv\" (UniqueName: \"kubernetes.io/projected/f7b097bb-1a53-48cc-8499-f13014278d10-kube-api-access-6lggv\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:13.166040 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.165910    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:13.166040 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.165915    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-workload-socket" pod "2abed568-6cc8-4336-b427-b8a976686c3f" (UID: "2abed568-6cc8-4336-b427-b8a976686c3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:13.166040 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.165926    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:13.166040 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.165941    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:13.166040 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.165953    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2abed568-6cc8-4336-b427-b8a976686c3f-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:13.166040 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.165967    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:13.166040 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.165979    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:13.166040 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.165993    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmjhs\" (UniqueName: \"kubernetes.io/projected/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-kube-api-access-kmjhs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:13.166040 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.166009    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7b097bb-1a53-48cc-8499-f13014278d10-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:13.166040 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.166024    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:13.166040 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.166037    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:13.166674 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.166051    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b097bb-1a53-48cc-8499-f13014278d10-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:13.166674 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.166066    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:13.166674 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.166079    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:13.166674 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.166092    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:13.166674 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.166109    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b097bb-1a53-48cc-8499-f13014278d10-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:13.166674 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.166122    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:13.166674 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.166136    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b097bb-1a53-48cc-8499-f13014278d10-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:13.167394 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.167367    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2abed568-6cc8-4336-b427-b8a976686c3f-kube-api-access-rvdtp" pod "2abed568-6cc8-4336-b427-b8a976686c3f" (UID: "2abed568-6cc8-4336-b427-b8a976686c3f"). InnerVolumeSpecName "kube-api-access-rvdtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:25:13.167544 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.167530    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-istio-envoy" pod "2abed568-6cc8-4336-b427-b8a976686c3f" (UID: "2abed568-6cc8-4336-b427-b8a976686c3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:13.167696 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.167670    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2abed568-6cc8-4336-b427-b8a976686c3f-istio-podinfo" pod "2abed568-6cc8-4336-b427-b8a976686c3f" (UID: "2abed568-6cc8-4336-b427-b8a976686c3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:25:13.167696 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.167674    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2abed568-6cc8-4336-b427-b8a976686c3f-istio-token" pod "2abed568-6cc8-4336-b427-b8a976686c3f" (UID: "2abed568-6cc8-4336-b427-b8a976686c3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:25:13.267055 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.267027    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvdtp\" (UniqueName: \"kubernetes.io/projected/2abed568-6cc8-4336-b427-b8a976686c3f-kube-api-access-rvdtp\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:13.267055 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.267051    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2abed568-6cc8-4336-b427-b8a976686c3f-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:13.267246 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.267063    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2abed568-6cc8-4336-b427-b8a976686c3f-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:13.267246 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.267074    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:13.267246 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.267082    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2abed568-6cc8-4336-b427-b8a976686c3f-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:13.409119 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.409087    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="570b97e3-1e23-474c-beb7-9438692c5508" path="/var/lib/kubelet/pods/570b97e3-1e23-474c-beb7-9438692c5508/volumes"
Oct 31 23:25:13.878543 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.878514    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"
Oct 31 23:25:13.879001 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.878538    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9" event={"ID":"f7b097bb-1a53-48cc-8499-f13014278d10","Type":"ContainerDied","Data":"b6974c19bb26558dbe0cb42c2c09ef9c7bcd108cd2344967a875996e853b223a"}
Oct 31 23:25:13.879001 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.878610    4301 scope.go:122] "RemoveContainer" containerID="0b1a39944e5d8230624e6db233fd50248f84730c3c5b5c9ded424d0a432d221f"
Oct 31 23:25:13.879904 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.879875    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw" event={"ID":"2abed568-6cc8-4336-b427-b8a976686c3f","Type":"ContainerDied","Data":"dbc5dee8664533599285b517eb4f4566ceaf682edd53a9b2d8f32c4fe20e6bb8"}
Oct 31 23:25:13.879904 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.879898    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"
Oct 31 23:25:13.881723 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.881704    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t" event={"ID":"fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5","Type":"ContainerDied","Data":"a7142c6baaaef9f5045860bc1fad336509b3c333ff1d32d0ee533ce079af8af5"}
Oct 31 23:25:13.881843 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.881746    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"
Oct 31 23:25:13.892112 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.892094    4301 scope.go:122] "RemoveContainer" containerID="23ab4029c0559ed24c259d479471c886a4ee16cfe31e79979cf99e43f8cf7bdc"
Oct 31 23:25:13.902235 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.902209    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"]
Oct 31 23:25:13.902526 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.902506    4301 scope.go:122] "RemoveContainer" containerID="5494c25f1a03d3cbc311c4194963bf23987c9df812cb86599e8fde7524ff4f2d"
Oct 31 23:25:13.906109 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.906087    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-mbqd9"]
Oct 31 23:25:13.918632 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.918610    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"]
Oct 31 23:25:13.925943 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.925920    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-7ckvw"]
Oct 31 23:25:13.935734 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.935709    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"]
Oct 31 23:25:13.944175 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:13.944150    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-57c9t"]
Oct 31 23:25:14.623308 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.623273    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:25:14.824535 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.813866    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-credential-socket\") pod \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") "
Oct 31 23:25:14.824535 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.813941    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istiod-ca-cert\") pod \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") "
Oct 31 23:25:14.824535 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.813979    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istio-envoy\") pod \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") "
Oct 31 23:25:14.824535 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.814027    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf839f5e-2570-44bd-9c62-7a65ea1590ed-kube-api-access-wzpvt\" (UniqueName: \"kubernetes.io/projected/bf839f5e-2570-44bd-9c62-7a65ea1590ed-kube-api-access-wzpvt\") pod \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") "
Oct 31 23:25:14.824535 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.814057    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-workload-socket\") pod \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") "
Oct 31 23:25:14.824535 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.814109    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istio-data\") pod \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") "
Oct 31 23:25:14.824535 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.814163    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istio-podinfo\") pod \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") "
Oct 31 23:25:14.824535 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.814242    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-workload-certs\") pod \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") "
Oct 31 23:25:14.824535 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.814406    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istio-token\" (UniqueName: \"kubernetes.io/projected/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istio-token\") pod \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\" (UID: \"bf839f5e-2570-44bd-9c62-7a65ea1590ed\") "
Oct 31 23:25:14.824535 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.815348    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-credential-socket" pod "bf839f5e-2570-44bd-9c62-7a65ea1590ed" (UID: "bf839f5e-2570-44bd-9c62-7a65ea1590ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:14.824535 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.815745    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istiod-ca-cert" pod "bf839f5e-2570-44bd-9c62-7a65ea1590ed" (UID: "bf839f5e-2570-44bd-9c62-7a65ea1590ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:25:14.824535 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.816453    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istio-data" pod "bf839f5e-2570-44bd-9c62-7a65ea1590ed" (UID: "bf839f5e-2570-44bd-9c62-7a65ea1590ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:14.824535 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.816634    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-workload-certs" pod "bf839f5e-2570-44bd-9c62-7a65ea1590ed" (UID: "bf839f5e-2570-44bd-9c62-7a65ea1590ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:14.824535 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.824491    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-workload-socket" pod "bf839f5e-2570-44bd-9c62-7a65ea1590ed" (UID: "bf839f5e-2570-44bd-9c62-7a65ea1590ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:14.827502 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.827457    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istio-envoy" pod "bf839f5e-2570-44bd-9c62-7a65ea1590ed" (UID: "bf839f5e-2570-44bd-9c62-7a65ea1590ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:14.828022 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.827988    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istio-podinfo" pod "bf839f5e-2570-44bd-9c62-7a65ea1590ed" (UID: "bf839f5e-2570-44bd-9c62-7a65ea1590ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:25:14.849463 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.849418    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf839f5e-2570-44bd-9c62-7a65ea1590ed-kube-api-access-wzpvt" pod "bf839f5e-2570-44bd-9c62-7a65ea1590ed" (UID: "bf839f5e-2570-44bd-9c62-7a65ea1590ed"). InnerVolumeSpecName "kube-api-access-wzpvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:25:14.850964 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.850907    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istio-token" pod "bf839f5e-2570-44bd-9c62-7a65ea1590ed" (UID: "bf839f5e-2570-44bd-9c62-7a65ea1590ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:25:14.918302 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.918256    4301 generic.go:356] "Generic (PLEG): container finished" podID="bf839f5e-2570-44bd-9c62-7a65ea1590ed" containerID="4706b538ebf255543d9badb25c61d0a5dc8efc72762b1e2dd71a58e52bc31c1e" exitCode=0
Oct 31 23:25:14.918995 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.918966    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2" event={"ID":"bf839f5e-2570-44bd-9c62-7a65ea1590ed","Type":"ContainerDied","Data":"4706b538ebf255543d9badb25c61d0a5dc8efc72762b1e2dd71a58e52bc31c1e"}
Oct 31 23:25:14.919217 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.919185    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2" event={"ID":"bf839f5e-2570-44bd-9c62-7a65ea1590ed","Type":"ContainerDied","Data":"2466c6f4b9280464d9cadb5d61823f7c3347f2e4d753bb01ef039172b8cd97f8"}
Oct 31 23:25:14.919399 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.919370    4301 scope.go:122] "RemoveContainer" containerID="4706b538ebf255543d9badb25c61d0a5dc8efc72762b1e2dd71a58e52bc31c1e"
Oct 31 23:25:14.919656 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.919625    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"
Oct 31 23:25:14.924456 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.924427    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:14.924572 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.924460    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:14.924572 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.924476    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:14.924572 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.924493    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzpvt\" (UniqueName: \"kubernetes.io/projected/bf839f5e-2570-44bd-9c62-7a65ea1590ed-kube-api-access-wzpvt\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:14.924572 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.924510    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:14.924572 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.924540    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:14.924572 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.924556    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:14.924572 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.924572    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf839f5e-2570-44bd-9c62-7a65ea1590ed-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:14.924977 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.924587    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf839f5e-2570-44bd-9c62-7a65ea1590ed-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:14.942967 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.938234    4301 generic.go:356] "Generic (PLEG): container finished" podID="d0195c12-01bd-4f4a-a72a-b608150516f2" containerID="b55f6e37daaf160cc50371071ce1abe701dcc03711497ddb0828c5bcba7adb06" exitCode=0
Oct 31 23:25:14.942967 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:14.942447    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg" event={"ID":"d0195c12-01bd-4f4a-a72a-b608150516f2","Type":"ContainerDied","Data":"b55f6e37daaf160cc50371071ce1abe701dcc03711497ddb0828c5bcba7adb06"}
Oct 31 23:25:15.131381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.130410    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"]
Oct 31 23:25:15.143382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.140715    4301 scope.go:122] "RemoveContainer" containerID="4706b538ebf255543d9badb25c61d0a5dc8efc72762b1e2dd71a58e52bc31c1e"
Oct 31 23:25:15.143382 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:25:15.141202    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4706b538ebf255543d9badb25c61d0a5dc8efc72762b1e2dd71a58e52bc31c1e\": container with ID starting with 4706b538ebf255543d9badb25c61d0a5dc8efc72762b1e2dd71a58e52bc31c1e not found: ID does not exist" containerID="4706b538ebf255543d9badb25c61d0a5dc8efc72762b1e2dd71a58e52bc31c1e"
Oct 31 23:25:15.143382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.141240    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4706b538ebf255543d9badb25c61d0a5dc8efc72762b1e2dd71a58e52bc31c1e"} err="failed to get container status \"4706b538ebf255543d9badb25c61d0a5dc8efc72762b1e2dd71a58e52bc31c1e\": rpc error: code = NotFound desc = could not find container \"4706b538ebf255543d9badb25c61d0a5dc8efc72762b1e2dd71a58e52bc31c1e\": container with ID starting with 4706b538ebf255543d9badb25c61d0a5dc8efc72762b1e2dd71a58e52bc31c1e not found: ID does not exist"
Oct 31 23:25:15.143382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.141267    4301 scope.go:122] "RemoveContainer" containerID="ef96b32c0a70a58bf905ba846040660f49d77848cc1a7acb6c6e6567c960e7b5"
Oct 31 23:25:15.167396 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.151515    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ql9g2"]
Oct 31 23:25:15.403659 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.403633    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2abed568-6cc8-4336-b427-b8a976686c3f" path="/var/lib/kubelet/pods/2abed568-6cc8-4336-b427-b8a976686c3f/volumes"
Oct 31 23:25:15.404143 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.404129    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf839f5e-2570-44bd-9c62-7a65ea1590ed" path="/var/lib/kubelet/pods/bf839f5e-2570-44bd-9c62-7a65ea1590ed/volumes"
Oct 31 23:25:15.404626 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.404613    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7b097bb-1a53-48cc-8499-f13014278d10" path="/var/lib/kubelet/pods/f7b097bb-1a53-48cc-8499-f13014278d10/volumes"
Oct 31 23:25:15.405395 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.405369    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5" path="/var/lib/kubelet/pods/fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5/volumes"
Oct 31 23:25:15.419997 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.419975    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:25:15.486947 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.486906    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0195c12-01bd-4f4a-a72a-b608150516f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0195c12-01bd-4f4a-a72a-b608150516f2-istiod-ca-cert\") pod \"d0195c12-01bd-4f4a-a72a-b608150516f2\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") "
Oct 31 23:25:15.487140 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.486976    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-credential-socket\") pod \"d0195c12-01bd-4f4a-a72a-b608150516f2\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") "
Oct 31 23:25:15.487140 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.487053    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0195c12-01bd-4f4a-a72a-b608150516f2-istio-token\" (UniqueName: \"kubernetes.io/projected/d0195c12-01bd-4f4a-a72a-b608150516f2-istio-token\") pod \"d0195c12-01bd-4f4a-a72a-b608150516f2\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") "
Oct 31 23:25:15.487140 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.487102    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0195c12-01bd-4f4a-a72a-b608150516f2-kube-api-access-pdbff\" (UniqueName: \"kubernetes.io/projected/d0195c12-01bd-4f4a-a72a-b608150516f2-kube-api-access-pdbff\") pod \"d0195c12-01bd-4f4a-a72a-b608150516f2\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") "
Oct 31 23:25:15.487331 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.487160    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-istio-envoy\") pod \"d0195c12-01bd-4f4a-a72a-b608150516f2\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") "
Oct 31 23:25:15.487331 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.487194    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-workload-socket\") pod \"d0195c12-01bd-4f4a-a72a-b608150516f2\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") "
Oct 31 23:25:15.487331 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.487227    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-workload-certs\") pod \"d0195c12-01bd-4f4a-a72a-b608150516f2\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") "
Oct 31 23:25:15.487536 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.487329    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0195c12-01bd-4f4a-a72a-b608150516f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0195c12-01bd-4f4a-a72a-b608150516f2-istio-podinfo\") pod \"d0195c12-01bd-4f4a-a72a-b608150516f2\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") "
Oct 31 23:25:15.487536 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.487376    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-credential-socket" pod "d0195c12-01bd-4f4a-a72a-b608150516f2" (UID: "d0195c12-01bd-4f4a-a72a-b608150516f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:15.487536 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.487408    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-istio-data\") pod \"d0195c12-01bd-4f4a-a72a-b608150516f2\" (UID: \"d0195c12-01bd-4f4a-a72a-b608150516f2\") "
Oct 31 23:25:15.487536 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.487421    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0195c12-01bd-4f4a-a72a-b608150516f2-istiod-ca-cert" pod "d0195c12-01bd-4f4a-a72a-b608150516f2" (UID: "d0195c12-01bd-4f4a-a72a-b608150516f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:25:15.487746 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.487613    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-workload-certs" pod "d0195c12-01bd-4f4a-a72a-b608150516f2" (UID: "d0195c12-01bd-4f4a-a72a-b608150516f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:15.487847 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.487812    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-workload-socket" pod "d0195c12-01bd-4f4a-a72a-b608150516f2" (UID: "d0195c12-01bd-4f4a-a72a-b608150516f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:15.487977 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.487846    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-istio-data" pod "d0195c12-01bd-4f4a-a72a-b608150516f2" (UID: "d0195c12-01bd-4f4a-a72a-b608150516f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:15.487977 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.487893    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:15.487977 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.487915    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0195c12-01bd-4f4a-a72a-b608150516f2-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:15.487977 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.487930    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:15.490094 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.490071    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-istio-envoy" pod "d0195c12-01bd-4f4a-a72a-b608150516f2" (UID: "d0195c12-01bd-4f4a-a72a-b608150516f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:15.490212 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.490097    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0195c12-01bd-4f4a-a72a-b608150516f2-istio-podinfo" pod "d0195c12-01bd-4f4a-a72a-b608150516f2" (UID: "d0195c12-01bd-4f4a-a72a-b608150516f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:25:15.490504 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.490486    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0195c12-01bd-4f4a-a72a-b608150516f2-kube-api-access-pdbff" pod "d0195c12-01bd-4f4a-a72a-b608150516f2" (UID: "d0195c12-01bd-4f4a-a72a-b608150516f2"). InnerVolumeSpecName "kube-api-access-pdbff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:25:15.490654 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.490629    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0195c12-01bd-4f4a-a72a-b608150516f2-istio-token" pod "d0195c12-01bd-4f4a-a72a-b608150516f2" (UID: "d0195c12-01bd-4f4a-a72a-b608150516f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:25:15.589104 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.589073    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:15.589104 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.589103    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0195c12-01bd-4f4a-a72a-b608150516f2-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:15.589464 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.589113    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdbff\" (UniqueName: \"kubernetes.io/projected/d0195c12-01bd-4f4a-a72a-b608150516f2-kube-api-access-pdbff\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:15.589464 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.589123    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:15.589464 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.589132    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0195c12-01bd-4f4a-a72a-b608150516f2-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:15.589464 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.589140    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0195c12-01bd-4f4a-a72a-b608150516f2-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:15.952304 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.952277    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"
Oct 31 23:25:15.952788 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.952270    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg" event={"ID":"d0195c12-01bd-4f4a-a72a-b608150516f2","Type":"ContainerDied","Data":"3fa7ae01589f39a9f78add93664f5b1ebf904cc781b0a3a59a0a92ba6db80331"}
Oct 31 23:25:15.952788 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.952436    4301 scope.go:122] "RemoveContainer" containerID="b55f6e37daaf160cc50371071ce1abe701dcc03711497ddb0828c5bcba7adb06"
Oct 31 23:25:15.974532 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.974508    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"]
Oct 31 23:25:15.981196 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:15.981175    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-m6dtg"]
Oct 31 23:25:17.377379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:17.373290    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"]
Oct 31 23:25:17.377379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:17.373607    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5" podUID="7a96c54a-c66d-466f-b2d4-84a208c3c664" containerName="istio-proxy" containerID="cri-o://44749bf5ac0653dd24caea25c2aae582b8055a50e26be94d07449f9f0ae54358" gracePeriod=30
Oct 31 23:25:17.418432 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:17.418392    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0195c12-01bd-4f4a-a72a-b608150516f2" path="/var/lib/kubelet/pods/d0195c12-01bd-4f4a-a72a-b608150516f2/volumes"
Oct 31 23:25:17.423035 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:17.423006    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"]
Oct 31 23:25:17.424192 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:17.424161    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl" podUID="92c87d0a-e7ea-44ed-a02d-c92554d9f41e" containerName="istio-proxy" containerID="cri-o://e39e6911cc519a121d4efa4339de5d4517d67781b9156b9d2ba4ad370a03dfa2" gracePeriod=30
Oct 31 23:25:17.426208 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:17.424475    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"]
Oct 31 23:25:17.426208 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:17.424631    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm" podUID="d361d757-5e4e-43a4-acb7-9ef5f24113af" containerName="istio-proxy" containerID="cri-o://8492f0084fcae522ce2b7c039dd6b8feba4000a065cae3671ef280a4218c4068" gracePeriod=30
Oct 31 23:25:18.456845 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:18.456811    4301 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 23:25:18.457268 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:18.456868    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5" podUID="7a96c54a-c66d-466f-b2d4-84a208c3c664" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 23:25:18.571185 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:18.571149    4301 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 23:25:18.571382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:18.571229    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl" podUID="92c87d0a-e7ea-44ed-a02d-c92554d9f41e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 23:25:18.817017 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:18.816938    4301 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 23:25:18.817017 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:18.816989    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm" podUID="d361d757-5e4e-43a4-acb7-9ef5f24113af" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 23:25:20.258161 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:20.258122    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"]
Oct 31 23:25:20.260407 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:20.260324    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g" podUID="85e9dd14-aa39-400e-b672-4c3122ea568a" containerName="istio-proxy" containerID="cri-o://febad0a1c9c7e1bd6fe226729255932abb68d1788e8ece571e9a507aceac5979" gracePeriod=30
Oct 31 23:25:22.079950 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.079915    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:25:22.708785 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.708749    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:25:22.792668 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.792618    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:25:22.796012 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.795992    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:25:22.897911 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.897875    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a96c54a-c66d-466f-b2d4-84a208c3c664-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a96c54a-c66d-466f-b2d4-84a208c3c664-istiod-ca-cert\") pod \"7a96c54a-c66d-466f-b2d4-84a208c3c664\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") "
Oct 31 23:25:22.898080 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.897928    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-kube-api-access-gkm4t\" (UniqueName: \"kubernetes.io/projected/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-kube-api-access-gkm4t\") pod \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") "
Oct 31 23:25:22.898080 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.897959    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istio-podinfo\") pod \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") "
Oct 31 23:25:22.898080 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.897995    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-workload-socket\") pod \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") "
Oct 31 23:25:22.898080 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.898023    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-istio-data\") pod \"d361d757-5e4e-43a4-acb7-9ef5f24113af\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") "
Oct 31 23:25:22.898435 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.898401    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-workload-socket" pod "92c87d0a-e7ea-44ed-a02d-c92554d9f41e" (UID: "92c87d0a-e7ea-44ed-a02d-c92554d9f41e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:22.898435 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.898414    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-istio-data" pod "d361d757-5e4e-43a4-acb7-9ef5f24113af" (UID: "d361d757-5e4e-43a4-acb7-9ef5f24113af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:22.898574 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.898427    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a96c54a-c66d-466f-b2d4-84a208c3c664-istiod-ca-cert" pod "7a96c54a-c66d-466f-b2d4-84a208c3c664" (UID: "7a96c54a-c66d-466f-b2d4-84a208c3c664"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:25:22.898574 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.898487    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-istio-data\") pod \"7a96c54a-c66d-466f-b2d4-84a208c3c664\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") "
Oct 31 23:25:22.898574 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.898544    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-workload-socket\") pod \"7a96c54a-c66d-466f-b2d4-84a208c3c664\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") "
Oct 31 23:25:22.898737 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.898579    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-credential-socket\") pod \"7a96c54a-c66d-466f-b2d4-84a208c3c664\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") "
Oct 31 23:25:22.898737 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.898610    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d361d757-5e4e-43a4-acb7-9ef5f24113af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d361d757-5e4e-43a4-acb7-9ef5f24113af-istio-podinfo\") pod \"d361d757-5e4e-43a4-acb7-9ef5f24113af\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") "
Oct 31 23:25:22.898737 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.898647    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d361d757-5e4e-43a4-acb7-9ef5f24113af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d361d757-5e4e-43a4-acb7-9ef5f24113af-istiod-ca-cert\") pod \"d361d757-5e4e-43a4-acb7-9ef5f24113af\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") "
Oct 31 23:25:22.898737 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.898683    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-workload-socket\") pod \"d361d757-5e4e-43a4-acb7-9ef5f24113af\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") "
Oct 31 23:25:22.898938 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.898768    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-istio-data" pod "7a96c54a-c66d-466f-b2d4-84a208c3c664" (UID: "7a96c54a-c66d-466f-b2d4-84a208c3c664"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:22.898938 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.898807    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istio-token\" (UniqueName: \"kubernetes.io/projected/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istio-token\") pod \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") "
Oct 31 23:25:22.898938 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.898860    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d361d757-5e4e-43a4-acb7-9ef5f24113af-istio-token\" (UniqueName: \"kubernetes.io/projected/d361d757-5e4e-43a4-acb7-9ef5f24113af-istio-token\") pod \"d361d757-5e4e-43a4-acb7-9ef5f24113af\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") "
Oct 31 23:25:22.898938 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.898890    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-istio-envoy\") pod \"7a96c54a-c66d-466f-b2d4-84a208c3c664\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") "
Oct 31 23:25:22.898938 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.898916    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-workload-certs\") pod \"7a96c54a-c66d-466f-b2d4-84a208c3c664\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") "
Oct 31 23:25:22.899204 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.898961    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a96c54a-c66d-466f-b2d4-84a208c3c664-kube-api-access-zjj5s\" (UniqueName: \"kubernetes.io/projected/7a96c54a-c66d-466f-b2d4-84a208c3c664-kube-api-access-zjj5s\") pod \"7a96c54a-c66d-466f-b2d4-84a208c3c664\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") "
Oct 31 23:25:22.899204 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.898978    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-credential-socket" pod "7a96c54a-c66d-466f-b2d4-84a208c3c664" (UID: "7a96c54a-c66d-466f-b2d4-84a208c3c664"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:22.899204 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.898993    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-workload-certs\") pod \"d361d757-5e4e-43a4-acb7-9ef5f24113af\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") "
Oct 31 23:25:22.899204 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.899051    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istio-data\") pod \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") "
Oct 31 23:25:22.899204 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.899085    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a96c54a-c66d-466f-b2d4-84a208c3c664-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a96c54a-c66d-466f-b2d4-84a208c3c664-istio-podinfo\") pod \"7a96c54a-c66d-466f-b2d4-84a208c3c664\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") "
Oct 31 23:25:22.899204 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.899112    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-credential-socket\") pod \"d361d757-5e4e-43a4-acb7-9ef5f24113af\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") "
Oct 31 23:25:22.899204 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.899139    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-credential-socket\") pod \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") "
Oct 31 23:25:22.899204 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.899170    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-workload-certs\") pod \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") "
Oct 31 23:25:22.899664 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.899208    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d361d757-5e4e-43a4-acb7-9ef5f24113af-kube-api-access-28cfg\" (UniqueName: \"kubernetes.io/projected/d361d757-5e4e-43a4-acb7-9ef5f24113af-kube-api-access-28cfg\") pod \"d361d757-5e4e-43a4-acb7-9ef5f24113af\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") "
Oct 31 23:25:22.899664 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.899241    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-istio-envoy\") pod \"d361d757-5e4e-43a4-acb7-9ef5f24113af\" (UID: \"d361d757-5e4e-43a4-acb7-9ef5f24113af\") "
Oct 31 23:25:22.899664 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.899279    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a96c54a-c66d-466f-b2d4-84a208c3c664-istio-token\" (UniqueName: \"kubernetes.io/projected/7a96c54a-c66d-466f-b2d4-84a208c3c664-istio-token\") pod \"7a96c54a-c66d-466f-b2d4-84a208c3c664\" (UID: \"7a96c54a-c66d-466f-b2d4-84a208c3c664\") "
Oct 31 23:25:22.899664 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.899305    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istio-envoy\") pod \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") "
Oct 31 23:25:22.899664 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.899350    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istiod-ca-cert\") pod \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\" (UID: \"92c87d0a-e7ea-44ed-a02d-c92554d9f41e\") "
Oct 31 23:25:22.899664 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.899618    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-workload-socket" pod "7a96c54a-c66d-466f-b2d4-84a208c3c664" (UID: "7a96c54a-c66d-466f-b2d4-84a208c3c664"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:22.899964 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.899932    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istiod-ca-cert" pod "92c87d0a-e7ea-44ed-a02d-c92554d9f41e" (UID: "92c87d0a-e7ea-44ed-a02d-c92554d9f41e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:25:22.902671 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.900008    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:22.902671 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.900051    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a96c54a-c66d-466f-b2d4-84a208c3c664-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:22.902671 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.900072    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:22.902671 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.900091    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:22.902671 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.900105    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:22.902671 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.900119    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:22.902671 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.900162    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-workload-certs" pod "7a96c54a-c66d-466f-b2d4-84a208c3c664" (UID: "7a96c54a-c66d-466f-b2d4-84a208c3c664"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:22.902671 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.900490    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d361d757-5e4e-43a4-acb7-9ef5f24113af-istiod-ca-cert" pod "d361d757-5e4e-43a4-acb7-9ef5f24113af" (UID: "d361d757-5e4e-43a4-acb7-9ef5f24113af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:25:22.902671 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.900793    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istio-podinfo" pod "92c87d0a-e7ea-44ed-a02d-c92554d9f41e" (UID: "92c87d0a-e7ea-44ed-a02d-c92554d9f41e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:25:22.902671 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.901040    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-kube-api-access-gkm4t" pod "92c87d0a-e7ea-44ed-a02d-c92554d9f41e" (UID: "92c87d0a-e7ea-44ed-a02d-c92554d9f41e"). InnerVolumeSpecName "kube-api-access-gkm4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:25:22.902671 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.901228    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-credential-socket" pod "92c87d0a-e7ea-44ed-a02d-c92554d9f41e" (UID: "92c87d0a-e7ea-44ed-a02d-c92554d9f41e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:22.902671 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.901553    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istio-data" pod "92c87d0a-e7ea-44ed-a02d-c92554d9f41e" (UID: "92c87d0a-e7ea-44ed-a02d-c92554d9f41e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:22.902671 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.901593    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-credential-socket" pod "d361d757-5e4e-43a4-acb7-9ef5f24113af" (UID: "d361d757-5e4e-43a4-acb7-9ef5f24113af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:22.902671 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.901806    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-workload-certs" pod "d361d757-5e4e-43a4-acb7-9ef5f24113af" (UID: "d361d757-5e4e-43a4-acb7-9ef5f24113af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:22.902671 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.902019    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-workload-certs" pod "92c87d0a-e7ea-44ed-a02d-c92554d9f41e" (UID: "92c87d0a-e7ea-44ed-a02d-c92554d9f41e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:22.902671 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.902073    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-istio-envoy" pod "7a96c54a-c66d-466f-b2d4-84a208c3c664" (UID: "7a96c54a-c66d-466f-b2d4-84a208c3c664"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:22.902671 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.902411    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-workload-socket" pod "d361d757-5e4e-43a4-acb7-9ef5f24113af" (UID: "d361d757-5e4e-43a4-acb7-9ef5f24113af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:22.902671 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.902622    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a96c54a-c66d-466f-b2d4-84a208c3c664-kube-api-access-zjj5s" pod "7a96c54a-c66d-466f-b2d4-84a208c3c664" (UID: "7a96c54a-c66d-466f-b2d4-84a208c3c664"). InnerVolumeSpecName "kube-api-access-zjj5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:25:22.903887 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.903534    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a96c54a-c66d-466f-b2d4-84a208c3c664-istio-podinfo" pod "7a96c54a-c66d-466f-b2d4-84a208c3c664" (UID: "7a96c54a-c66d-466f-b2d4-84a208c3c664"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:25:22.903887 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.903820    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-istio-envoy" pod "d361d757-5e4e-43a4-acb7-9ef5f24113af" (UID: "d361d757-5e4e-43a4-acb7-9ef5f24113af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:22.904400 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.904295    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d361d757-5e4e-43a4-acb7-9ef5f24113af-istio-token" pod "d361d757-5e4e-43a4-acb7-9ef5f24113af" (UID: "d361d757-5e4e-43a4-acb7-9ef5f24113af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:25:22.904400 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.904323    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istio-token" pod "92c87d0a-e7ea-44ed-a02d-c92554d9f41e" (UID: "92c87d0a-e7ea-44ed-a02d-c92554d9f41e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:25:22.904847 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.904785    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d361d757-5e4e-43a4-acb7-9ef5f24113af-kube-api-access-28cfg" pod "d361d757-5e4e-43a4-acb7-9ef5f24113af" (UID: "d361d757-5e4e-43a4-acb7-9ef5f24113af"). InnerVolumeSpecName "kube-api-access-28cfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:25:22.904938 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.904912    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istio-envoy" pod "92c87d0a-e7ea-44ed-a02d-c92554d9f41e" (UID: "92c87d0a-e7ea-44ed-a02d-c92554d9f41e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:22.905459 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.905412    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d361d757-5e4e-43a4-acb7-9ef5f24113af-istio-podinfo" pod "d361d757-5e4e-43a4-acb7-9ef5f24113af" (UID: "d361d757-5e4e-43a4-acb7-9ef5f24113af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:25:22.905801 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:22.905782    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a96c54a-c66d-466f-b2d4-84a208c3c664-istio-token" pod "7a96c54a-c66d-466f-b2d4-84a208c3c664" (UID: "7a96c54a-c66d-466f-b2d4-84a208c3c664"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:25:23.001676 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.001557    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkm4t\" (UniqueName: \"kubernetes.io/projected/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-kube-api-access-gkm4t\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:23.001676 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.001615    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:23.001676 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.001630    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d361d757-5e4e-43a4-acb7-9ef5f24113af-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:23.001676 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.001644    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d361d757-5e4e-43a4-acb7-9ef5f24113af-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:23.001676 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.001656    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:23.001676 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.001668    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:23.001676 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.001681    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d361d757-5e4e-43a4-acb7-9ef5f24113af-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:23.002054 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.001694    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:23.002054 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.001705    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a96c54a-c66d-466f-b2d4-84a208c3c664-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:23.002054 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.001717    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjj5s\" (UniqueName: \"kubernetes.io/projected/7a96c54a-c66d-466f-b2d4-84a208c3c664-kube-api-access-zjj5s\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:23.002054 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.001728    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:23.002054 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.001741    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:23.002054 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.001752    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a96c54a-c66d-466f-b2d4-84a208c3c664-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:23.002054 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.001765    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:23.002054 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.001776    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:23.002054 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.001788    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:23.002054 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.001801    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28cfg\" (UniqueName: \"kubernetes.io/projected/d361d757-5e4e-43a4-acb7-9ef5f24113af-kube-api-access-28cfg\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:23.002054 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.001813    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d361d757-5e4e-43a4-acb7-9ef5f24113af-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:23.002054 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.001826    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a96c54a-c66d-466f-b2d4-84a208c3c664-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:23.002054 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.001837    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:23.002054 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.001850    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c87d0a-e7ea-44ed-a02d-c92554d9f41e-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:23.024959 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.024922    4301 generic.go:356] "Generic (PLEG): container finished" podID="d361d757-5e4e-43a4-acb7-9ef5f24113af" containerID="8492f0084fcae522ce2b7c039dd6b8feba4000a065cae3671ef280a4218c4068" exitCode=0
Oct 31 23:25:23.025196 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.024973    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"
Oct 31 23:25:23.025196 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.025008    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm" event={"ID":"d361d757-5e4e-43a4-acb7-9ef5f24113af","Type":"ContainerDied","Data":"8492f0084fcae522ce2b7c039dd6b8feba4000a065cae3671ef280a4218c4068"}
Oct 31 23:25:23.025196 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.025073    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm" event={"ID":"d361d757-5e4e-43a4-acb7-9ef5f24113af","Type":"ContainerDied","Data":"a400b386ef625053d3ae414a6f86fd02b36eb663691a2b826940a12f08b9944f"}
Oct 31 23:25:23.025196 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.025092    4301 scope.go:122] "RemoveContainer" containerID="8492f0084fcae522ce2b7c039dd6b8feba4000a065cae3671ef280a4218c4068"
Oct 31 23:25:23.026606 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.026580    4301 generic.go:356] "Generic (PLEG): container finished" podID="7a96c54a-c66d-466f-b2d4-84a208c3c664" containerID="44749bf5ac0653dd24caea25c2aae582b8055a50e26be94d07449f9f0ae54358" exitCode=0
Oct 31 23:25:23.026732 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.026634    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"
Oct 31 23:25:23.026732 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.026658    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5" event={"ID":"7a96c54a-c66d-466f-b2d4-84a208c3c664","Type":"ContainerDied","Data":"44749bf5ac0653dd24caea25c2aae582b8055a50e26be94d07449f9f0ae54358"}
Oct 31 23:25:23.026732 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.026705    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5" event={"ID":"7a96c54a-c66d-466f-b2d4-84a208c3c664","Type":"ContainerDied","Data":"8d7d8e2468f5286ba022818ca4c1b081230f64db297bdc28be77ca75954f9b75"}
Oct 31 23:25:23.028435 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.028410    4301 generic.go:356] "Generic (PLEG): container finished" podID="92c87d0a-e7ea-44ed-a02d-c92554d9f41e" containerID="e39e6911cc519a121d4efa4339de5d4517d67781b9156b9d2ba4ad370a03dfa2" exitCode=0
Oct 31 23:25:23.028560 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.028495    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"
Oct 31 23:25:23.028560 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.028543    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl" event={"ID":"92c87d0a-e7ea-44ed-a02d-c92554d9f41e","Type":"ContainerDied","Data":"e39e6911cc519a121d4efa4339de5d4517d67781b9156b9d2ba4ad370a03dfa2"}
Oct 31 23:25:23.028679 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.028578    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl" event={"ID":"92c87d0a-e7ea-44ed-a02d-c92554d9f41e","Type":"ContainerDied","Data":"40df41ab770e7201fbc719db54c127663a3f2ef6b8c5a3a09c1a7ce9c52e0c06"}
Oct 31 23:25:23.037207 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.037184    4301 scope.go:122] "RemoveContainer" containerID="8492f0084fcae522ce2b7c039dd6b8feba4000a065cae3671ef280a4218c4068"
Oct 31 23:25:23.037577 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:25:23.037559    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8492f0084fcae522ce2b7c039dd6b8feba4000a065cae3671ef280a4218c4068\": container with ID starting with 8492f0084fcae522ce2b7c039dd6b8feba4000a065cae3671ef280a4218c4068 not found: ID does not exist" containerID="8492f0084fcae522ce2b7c039dd6b8feba4000a065cae3671ef280a4218c4068"
Oct 31 23:25:23.037651 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.037585    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8492f0084fcae522ce2b7c039dd6b8feba4000a065cae3671ef280a4218c4068"} err="failed to get container status \"8492f0084fcae522ce2b7c039dd6b8feba4000a065cae3671ef280a4218c4068\": rpc error: code = NotFound desc = could not find container \"8492f0084fcae522ce2b7c039dd6b8feba4000a065cae3671ef280a4218c4068\": container with ID starting with 8492f0084fcae522ce2b7c039dd6b8feba4000a065cae3671ef280a4218c4068 not found: ID does not exist"
Oct 31 23:25:23.037651 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.037616    4301 scope.go:122] "RemoveContainer" containerID="44749bf5ac0653dd24caea25c2aae582b8055a50e26be94d07449f9f0ae54358"
Oct 31 23:25:23.046952 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.046933    4301 scope.go:122] "RemoveContainer" containerID="44749bf5ac0653dd24caea25c2aae582b8055a50e26be94d07449f9f0ae54358"
Oct 31 23:25:23.047228 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:25:23.047207    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44749bf5ac0653dd24caea25c2aae582b8055a50e26be94d07449f9f0ae54358\": container with ID starting with 44749bf5ac0653dd24caea25c2aae582b8055a50e26be94d07449f9f0ae54358 not found: ID does not exist" containerID="44749bf5ac0653dd24caea25c2aae582b8055a50e26be94d07449f9f0ae54358"
Oct 31 23:25:23.047303 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.047238    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44749bf5ac0653dd24caea25c2aae582b8055a50e26be94d07449f9f0ae54358"} err="failed to get container status \"44749bf5ac0653dd24caea25c2aae582b8055a50e26be94d07449f9f0ae54358\": rpc error: code = NotFound desc = could not find container \"44749bf5ac0653dd24caea25c2aae582b8055a50e26be94d07449f9f0ae54358\": container with ID starting with 44749bf5ac0653dd24caea25c2aae582b8055a50e26be94d07449f9f0ae54358 not found: ID does not exist"
Oct 31 23:25:23.047303 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.047262    4301 scope.go:122] "RemoveContainer" containerID="e39e6911cc519a121d4efa4339de5d4517d67781b9156b9d2ba4ad370a03dfa2"
Oct 31 23:25:23.051629 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.051606    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"]
Oct 31 23:25:23.057873 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.057849    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-nnfdm"]
Oct 31 23:25:23.058939 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.058902    4301 scope.go:122] "RemoveContainer" containerID="e39e6911cc519a121d4efa4339de5d4517d67781b9156b9d2ba4ad370a03dfa2"
Oct 31 23:25:23.059250 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:25:23.059222    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e39e6911cc519a121d4efa4339de5d4517d67781b9156b9d2ba4ad370a03dfa2\": container with ID starting with e39e6911cc519a121d4efa4339de5d4517d67781b9156b9d2ba4ad370a03dfa2 not found: ID does not exist" containerID="e39e6911cc519a121d4efa4339de5d4517d67781b9156b9d2ba4ad370a03dfa2"
Oct 31 23:25:23.059328 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.059253    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e39e6911cc519a121d4efa4339de5d4517d67781b9156b9d2ba4ad370a03dfa2"} err="failed to get container status \"e39e6911cc519a121d4efa4339de5d4517d67781b9156b9d2ba4ad370a03dfa2\": rpc error: code = NotFound desc = could not find container \"e39e6911cc519a121d4efa4339de5d4517d67781b9156b9d2ba4ad370a03dfa2\": container with ID starting with e39e6911cc519a121d4efa4339de5d4517d67781b9156b9d2ba4ad370a03dfa2 not found: ID does not exist"
Oct 31 23:25:23.068595 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.068574    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"]
Oct 31 23:25:23.074245 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.074225    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-tlkxl"]
Oct 31 23:25:23.093843 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.093819    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"]
Oct 31 23:25:23.097461 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.097432    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-klkt5"]
Oct 31 23:25:23.345716 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.345633    4301 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 23:25:23.345716 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.345701    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g" podUID="85e9dd14-aa39-400e-b672-4c3122ea568a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 23:25:23.404370 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.404320    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a96c54a-c66d-466f-b2d4-84a208c3c664" path="/var/lib/kubelet/pods/7a96c54a-c66d-466f-b2d4-84a208c3c664/volumes"
Oct 31 23:25:23.404955 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.404925    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92c87d0a-e7ea-44ed-a02d-c92554d9f41e" path="/var/lib/kubelet/pods/92c87d0a-e7ea-44ed-a02d-c92554d9f41e/volumes"
Oct 31 23:25:23.405416 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:23.405399    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d361d757-5e4e-43a4-acb7-9ef5f24113af" path="/var/lib/kubelet/pods/d361d757-5e4e-43a4-acb7-9ef5f24113af/volumes"
Oct 31 23:25:25.473433 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:25.472671    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"]
Oct 31 23:25:25.473433 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:25.472926    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr" podUID="195909b1-e0bf-4679-84e3-50786cc7eaea" containerName="istio-proxy" containerID="cri-o://2a1750e13aa82eed80f46e0007e854c4b42f473c1b9ac4faaa1720815e3015d4" gracePeriod=30
Oct 31 23:25:25.531599 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:25.531569    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:25:25.632863 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:25.632818    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-credential-socket\") pod \"85e9dd14-aa39-400e-b672-4c3122ea568a\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") "
Oct 31 23:25:25.633062 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:25.632890    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85e9dd14-aa39-400e-b672-4c3122ea568a-istio-token\" (UniqueName: \"kubernetes.io/projected/85e9dd14-aa39-400e-b672-4c3122ea568a-istio-token\") pod \"85e9dd14-aa39-400e-b672-4c3122ea568a\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") "
Oct 31 23:25:25.633062 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:25.632975    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-istio-envoy\") pod \"85e9dd14-aa39-400e-b672-4c3122ea568a\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") "
Oct 31 23:25:25.633062 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:25.633023    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85e9dd14-aa39-400e-b672-4c3122ea568a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85e9dd14-aa39-400e-b672-4c3122ea568a-istio-podinfo\") pod \"85e9dd14-aa39-400e-b672-4c3122ea568a\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") "
Oct 31 23:25:25.633062 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:25.633057    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85e9dd14-aa39-400e-b672-4c3122ea568a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85e9dd14-aa39-400e-b672-4c3122ea568a-istiod-ca-cert\") pod \"85e9dd14-aa39-400e-b672-4c3122ea568a\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") "
Oct 31 23:25:25.633328 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:25.633111    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-workload-certs\") pod \"85e9dd14-aa39-400e-b672-4c3122ea568a\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") "
Oct 31 23:25:25.633328 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:25.633147    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85e9dd14-aa39-400e-b672-4c3122ea568a-kube-api-access-7j5w8\" (UniqueName: \"kubernetes.io/projected/85e9dd14-aa39-400e-b672-4c3122ea568a-kube-api-access-7j5w8\") pod \"85e9dd14-aa39-400e-b672-4c3122ea568a\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") "
Oct 31 23:25:25.633328 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:25.633201    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-istio-data\") pod \"85e9dd14-aa39-400e-b672-4c3122ea568a\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") "
Oct 31 23:25:25.633328 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:25.633230    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-workload-socket\") pod \"85e9dd14-aa39-400e-b672-4c3122ea568a\" (UID: \"85e9dd14-aa39-400e-b672-4c3122ea568a\") "
Oct 31 23:25:25.636185 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:25.634894    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85e9dd14-aa39-400e-b672-4c3122ea568a-istiod-ca-cert" pod "85e9dd14-aa39-400e-b672-4c3122ea568a" (UID: "85e9dd14-aa39-400e-b672-4c3122ea568a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:25:25.636185 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:25.635156    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-istio-data" pod "85e9dd14-aa39-400e-b672-4c3122ea568a" (UID: "85e9dd14-aa39-400e-b672-4c3122ea568a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:25.637223 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:25.637194    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-workload-certs" pod "85e9dd14-aa39-400e-b672-4c3122ea568a" (UID: "85e9dd14-aa39-400e-b672-4c3122ea568a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:25.638961 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:25.637437    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-credential-socket" pod "85e9dd14-aa39-400e-b672-4c3122ea568a" (UID: "85e9dd14-aa39-400e-b672-4c3122ea568a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:25.638961 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:25.637500    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-workload-socket" pod "85e9dd14-aa39-400e-b672-4c3122ea568a" (UID: "85e9dd14-aa39-400e-b672-4c3122ea568a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:25.639823 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:25.639781    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85e9dd14-aa39-400e-b672-4c3122ea568a-kube-api-access-7j5w8" pod "85e9dd14-aa39-400e-b672-4c3122ea568a" (UID: "85e9dd14-aa39-400e-b672-4c3122ea568a"). InnerVolumeSpecName "kube-api-access-7j5w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:25:25.640950 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:25.640452    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-istio-envoy" pod "85e9dd14-aa39-400e-b672-4c3122ea568a" (UID: "85e9dd14-aa39-400e-b672-4c3122ea568a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:25.640950 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:25.640709    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85e9dd14-aa39-400e-b672-4c3122ea568a-istio-podinfo" pod "85e9dd14-aa39-400e-b672-4c3122ea568a" (UID: "85e9dd14-aa39-400e-b672-4c3122ea568a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:25:25.641467 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:25.641437    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85e9dd14-aa39-400e-b672-4c3122ea568a-istio-token" pod "85e9dd14-aa39-400e-b672-4c3122ea568a" (UID: "85e9dd14-aa39-400e-b672-4c3122ea568a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:25:25.734051 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:25.734017    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:25.734051 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:25.734046    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85e9dd14-aa39-400e-b672-4c3122ea568a-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:25.734051 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:25.734056    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:25.734304 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:25.734065    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85e9dd14-aa39-400e-b672-4c3122ea568a-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:25.734304 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:25.734074    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85e9dd14-aa39-400e-b672-4c3122ea568a-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:25.734304 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:25.734082    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:25.734304 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:25.734091    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j5w8\" (UniqueName: \"kubernetes.io/projected/85e9dd14-aa39-400e-b672-4c3122ea568a-kube-api-access-7j5w8\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:25.734304 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:25.734118    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:25.734304 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:25.734126    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e9dd14-aa39-400e-b672-4c3122ea568a-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:26.086458 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:26.085544    4301 generic.go:356] "Generic (PLEG): container finished" podID="85e9dd14-aa39-400e-b672-4c3122ea568a" containerID="febad0a1c9c7e1bd6fe226729255932abb68d1788e8ece571e9a507aceac5979" exitCode=0
Oct 31 23:25:26.086458 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:26.085679    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"
Oct 31 23:25:26.086458 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:26.085715    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g" event={"ID":"85e9dd14-aa39-400e-b672-4c3122ea568a","Type":"ContainerDied","Data":"febad0a1c9c7e1bd6fe226729255932abb68d1788e8ece571e9a507aceac5979"}
Oct 31 23:25:26.086458 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:26.085799    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g" event={"ID":"85e9dd14-aa39-400e-b672-4c3122ea568a","Type":"ContainerDied","Data":"7c5bf25d497937f98f53c562d7ff56f91928b68a630de341ff323896e3c04505"}
Oct 31 23:25:26.086458 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:26.085828    4301 scope.go:122] "RemoveContainer" containerID="febad0a1c9c7e1bd6fe226729255932abb68d1788e8ece571e9a507aceac5979"
Oct 31 23:25:26.098519 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:26.098498    4301 scope.go:122] "RemoveContainer" containerID="febad0a1c9c7e1bd6fe226729255932abb68d1788e8ece571e9a507aceac5979"
Oct 31 23:25:26.098831 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:25:26.098812    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"febad0a1c9c7e1bd6fe226729255932abb68d1788e8ece571e9a507aceac5979\": container with ID starting with febad0a1c9c7e1bd6fe226729255932abb68d1788e8ece571e9a507aceac5979 not found: ID does not exist" containerID="febad0a1c9c7e1bd6fe226729255932abb68d1788e8ece571e9a507aceac5979"
Oct 31 23:25:26.098901 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:26.098841    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"febad0a1c9c7e1bd6fe226729255932abb68d1788e8ece571e9a507aceac5979"} err="failed to get container status \"febad0a1c9c7e1bd6fe226729255932abb68d1788e8ece571e9a507aceac5979\": rpc error: code = NotFound desc = could not find container \"febad0a1c9c7e1bd6fe226729255932abb68d1788e8ece571e9a507aceac5979\": container with ID starting with febad0a1c9c7e1bd6fe226729255932abb68d1788e8ece571e9a507aceac5979 not found: ID does not exist"
Oct 31 23:25:26.110768 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:26.110735    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"]
Oct 31 23:25:26.119130 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:26.119106    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-q768g"]
Oct 31 23:25:27.402626 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:27.402594    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85e9dd14-aa39-400e-b672-4c3122ea568a" path="/var/lib/kubelet/pods/85e9dd14-aa39-400e-b672-4c3122ea568a/volumes"
Oct 31 23:25:30.731701 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:30.731672    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:25:30.790950 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:30.790871    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-istio-data\") pod \"195909b1-e0bf-4679-84e3-50786cc7eaea\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") "
Oct 31 23:25:30.790950 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:30.790912    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-credential-socket\") pod \"195909b1-e0bf-4679-84e3-50786cc7eaea\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") "
Oct 31 23:25:30.791174 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:30.790955    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/195909b1-e0bf-4679-84e3-50786cc7eaea-istio-token\" (UniqueName: \"kubernetes.io/projected/195909b1-e0bf-4679-84e3-50786cc7eaea-istio-token\") pod \"195909b1-e0bf-4679-84e3-50786cc7eaea\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") "
Oct 31 23:25:30.791174 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:30.791009    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/195909b1-e0bf-4679-84e3-50786cc7eaea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/195909b1-e0bf-4679-84e3-50786cc7eaea-istio-podinfo\") pod \"195909b1-e0bf-4679-84e3-50786cc7eaea\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") "
Oct 31 23:25:30.791174 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:30.791052    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-istio-envoy\") pod \"195909b1-e0bf-4679-84e3-50786cc7eaea\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") "
Oct 31 23:25:30.791174 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:30.791147    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-workload-certs\") pod \"195909b1-e0bf-4679-84e3-50786cc7eaea\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") "
Oct 31 23:25:30.791483 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:30.791178    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/195909b1-e0bf-4679-84e3-50786cc7eaea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/195909b1-e0bf-4679-84e3-50786cc7eaea-istiod-ca-cert\") pod \"195909b1-e0bf-4679-84e3-50786cc7eaea\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") "
Oct 31 23:25:30.791483 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:30.791212    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-istio-data" pod "195909b1-e0bf-4679-84e3-50786cc7eaea" (UID: "195909b1-e0bf-4679-84e3-50786cc7eaea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:30.791483 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:30.791239    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-credential-socket" pod "195909b1-e0bf-4679-84e3-50786cc7eaea" (UID: "195909b1-e0bf-4679-84e3-50786cc7eaea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:30.791483 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:30.791247    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/195909b1-e0bf-4679-84e3-50786cc7eaea-kube-api-access-v72j5\" (UniqueName: \"kubernetes.io/projected/195909b1-e0bf-4679-84e3-50786cc7eaea-kube-api-access-v72j5\") pod \"195909b1-e0bf-4679-84e3-50786cc7eaea\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") "
Oct 31 23:25:30.791483 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:30.791314    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-workload-socket\") pod \"195909b1-e0bf-4679-84e3-50786cc7eaea\" (UID: \"195909b1-e0bf-4679-84e3-50786cc7eaea\") "
Oct 31 23:25:30.791483 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:30.791441    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-workload-certs" pod "195909b1-e0bf-4679-84e3-50786cc7eaea" (UID: "195909b1-e0bf-4679-84e3-50786cc7eaea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:30.791764 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:30.791624    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/195909b1-e0bf-4679-84e3-50786cc7eaea-istiod-ca-cert" pod "195909b1-e0bf-4679-84e3-50786cc7eaea" (UID: "195909b1-e0bf-4679-84e3-50786cc7eaea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:25:30.791764 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:30.791749    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-workload-socket" pod "195909b1-e0bf-4679-84e3-50786cc7eaea" (UID: "195909b1-e0bf-4679-84e3-50786cc7eaea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:30.791945 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:30.791901    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:30.791945 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:30.791928    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:30.791945 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:30.791943    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:30.792057 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:30.791956    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:30.792057 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:30.791970    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/195909b1-e0bf-4679-84e3-50786cc7eaea-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:30.793397 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:30.793366    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/195909b1-e0bf-4679-84e3-50786cc7eaea-istio-podinfo" pod "195909b1-e0bf-4679-84e3-50786cc7eaea" (UID: "195909b1-e0bf-4679-84e3-50786cc7eaea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:25:30.793493 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:30.793389    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/195909b1-e0bf-4679-84e3-50786cc7eaea-kube-api-access-v72j5" pod "195909b1-e0bf-4679-84e3-50786cc7eaea" (UID: "195909b1-e0bf-4679-84e3-50786cc7eaea"). InnerVolumeSpecName "kube-api-access-v72j5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:25:30.793716 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:30.793695    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-istio-envoy" pod "195909b1-e0bf-4679-84e3-50786cc7eaea" (UID: "195909b1-e0bf-4679-84e3-50786cc7eaea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:30.793755 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:30.793708    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/195909b1-e0bf-4679-84e3-50786cc7eaea-istio-token" pod "195909b1-e0bf-4679-84e3-50786cc7eaea" (UID: "195909b1-e0bf-4679-84e3-50786cc7eaea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:25:30.892790 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:30.892751    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/195909b1-e0bf-4679-84e3-50786cc7eaea-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:30.892790 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:30.892783    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/195909b1-e0bf-4679-84e3-50786cc7eaea-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:30.892790 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:30.892792    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v72j5\" (UniqueName: \"kubernetes.io/projected/195909b1-e0bf-4679-84e3-50786cc7eaea-kube-api-access-v72j5\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:30.893029 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:30.892803    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/195909b1-e0bf-4679-84e3-50786cc7eaea-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:30.998201 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:30.998163    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"]
Oct 31 23:25:31.123035 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:31.122999    4301 generic.go:356] "Generic (PLEG): container finished" podID="195909b1-e0bf-4679-84e3-50786cc7eaea" containerID="2a1750e13aa82eed80f46e0007e854c4b42f473c1b9ac4faaa1720815e3015d4" exitCode=0
Oct 31 23:25:31.123208 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:31.123059    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"
Oct 31 23:25:31.123208 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:31.123087    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr" event={"ID":"195909b1-e0bf-4679-84e3-50786cc7eaea","Type":"ContainerDied","Data":"2a1750e13aa82eed80f46e0007e854c4b42f473c1b9ac4faaa1720815e3015d4"}
Oct 31 23:25:31.123208 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:31.123139    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr" event={"ID":"195909b1-e0bf-4679-84e3-50786cc7eaea","Type":"ContainerDied","Data":"3bf10819693fdbd41d12aa599ed252707774c868117c04fa39c9f2f9eb42575a"}
Oct 31 23:25:31.123208 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:31.123165    4301 scope.go:122] "RemoveContainer" containerID="2a1750e13aa82eed80f46e0007e854c4b42f473c1b9ac4faaa1720815e3015d4"
Oct 31 23:25:31.133917 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:31.133901    4301 scope.go:122] "RemoveContainer" containerID="f8568961287ae2725915895a8347a54e94c47cc3e4bba476fb3f355ef5c291e8"
Oct 31 23:25:31.142461 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:31.142445    4301 scope.go:122] "RemoveContainer" containerID="2a1750e13aa82eed80f46e0007e854c4b42f473c1b9ac4faaa1720815e3015d4"
Oct 31 23:25:31.142719 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:25:31.142701    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a1750e13aa82eed80f46e0007e854c4b42f473c1b9ac4faaa1720815e3015d4\": container with ID starting with 2a1750e13aa82eed80f46e0007e854c4b42f473c1b9ac4faaa1720815e3015d4 not found: ID does not exist" containerID="2a1750e13aa82eed80f46e0007e854c4b42f473c1b9ac4faaa1720815e3015d4"
Oct 31 23:25:31.142775 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:31.142728    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a1750e13aa82eed80f46e0007e854c4b42f473c1b9ac4faaa1720815e3015d4"} err="failed to get container status \"2a1750e13aa82eed80f46e0007e854c4b42f473c1b9ac4faaa1720815e3015d4\": rpc error: code = NotFound desc = could not find container \"2a1750e13aa82eed80f46e0007e854c4b42f473c1b9ac4faaa1720815e3015d4\": container with ID starting with 2a1750e13aa82eed80f46e0007e854c4b42f473c1b9ac4faaa1720815e3015d4 not found: ID does not exist"
Oct 31 23:25:31.142775 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:31.142749    4301 scope.go:122] "RemoveContainer" containerID="f8568961287ae2725915895a8347a54e94c47cc3e4bba476fb3f355ef5c291e8"
Oct 31 23:25:31.142955 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:25:31.142937    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8568961287ae2725915895a8347a54e94c47cc3e4bba476fb3f355ef5c291e8\": container with ID starting with f8568961287ae2725915895a8347a54e94c47cc3e4bba476fb3f355ef5c291e8 not found: ID does not exist" containerID="f8568961287ae2725915895a8347a54e94c47cc3e4bba476fb3f355ef5c291e8"
Oct 31 23:25:31.143045 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:31.142956    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8568961287ae2725915895a8347a54e94c47cc3e4bba476fb3f355ef5c291e8"} err="failed to get container status \"f8568961287ae2725915895a8347a54e94c47cc3e4bba476fb3f355ef5c291e8\": rpc error: code = NotFound desc = could not find container \"f8568961287ae2725915895a8347a54e94c47cc3e4bba476fb3f355ef5c291e8\": container with ID starting with f8568961287ae2725915895a8347a54e94c47cc3e4bba476fb3f355ef5c291e8 not found: ID does not exist"
Oct 31 23:25:31.151922 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:31.151889    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"]
Oct 31 23:25:31.152138 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:31.152124    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9e90c-openshift-default-5f9bdd966f-xtmhr"]
Oct 31 23:25:31.403349 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:31.403272    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="195909b1-e0bf-4679-84e3-50786cc7eaea" path="/var/lib/kubelet/pods/195909b1-e0bf-4679-84e3-50786cc7eaea/volumes"
Oct 31 23:25:32.545501 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:25:32.545467    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:25:46.018150 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:46.018094    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main" containerID="cri-o://ad356fb1ac3c6dbac69dd222c782823e58db0471c2948d13dd7f17193a8de41b" gracePeriod=45
Oct 31 23:25:46.224841 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:46.224810    4301 generic.go:356] "Generic (PLEG): container finished" podID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerID="ad356fb1ac3c6dbac69dd222c782823e58db0471c2948d13dd7f17193a8de41b" exitCode=0
Oct 31 23:25:46.225082 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:46.224938    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" event={"ID":"b1cc0440-3a85-456d-981a-915ae2f4e0bb","Type":"ContainerDied","Data":"ad356fb1ac3c6dbac69dd222c782823e58db0471c2948d13dd7f17193a8de41b"}
Oct 31 23:25:46.278973 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:46.278906    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:25:46.364465 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:46.364424    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1cc0440-3a85-456d-981a-915ae2f4e0bb-tls-certs\" (UniqueName: \"kubernetes.io/secret/b1cc0440-3a85-456d-981a-915ae2f4e0bb-tls-certs\") pod \"b1cc0440-3a85-456d-981a-915ae2f4e0bb\" (UID: \"b1cc0440-3a85-456d-981a-915ae2f4e0bb\") "
Oct 31 23:25:46.364649 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:46.364552    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1cc0440-3a85-456d-981a-915ae2f4e0bb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1cc0440-3a85-456d-981a-915ae2f4e0bb-dshm\") pod \"b1cc0440-3a85-456d-981a-915ae2f4e0bb\" (UID: \"b1cc0440-3a85-456d-981a-915ae2f4e0bb\") "
Oct 31 23:25:46.364649 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:46.364606    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1cc0440-3a85-456d-981a-915ae2f4e0bb-home\" (UniqueName: \"kubernetes.io/empty-dir/b1cc0440-3a85-456d-981a-915ae2f4e0bb-home\") pod \"b1cc0440-3a85-456d-981a-915ae2f4e0bb\" (UID: \"b1cc0440-3a85-456d-981a-915ae2f4e0bb\") "
Oct 31 23:25:46.364764 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:46.364688    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1cc0440-3a85-456d-981a-915ae2f4e0bb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1cc0440-3a85-456d-981a-915ae2f4e0bb-model-cache\") pod \"b1cc0440-3a85-456d-981a-915ae2f4e0bb\" (UID: \"b1cc0440-3a85-456d-981a-915ae2f4e0bb\") "
Oct 31 23:25:46.364826 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:46.364766    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1cc0440-3a85-456d-981a-915ae2f4e0bb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1cc0440-3a85-456d-981a-915ae2f4e0bb-tmp-dir\") pod \"b1cc0440-3a85-456d-981a-915ae2f4e0bb\" (UID: \"b1cc0440-3a85-456d-981a-915ae2f4e0bb\") "
Oct 31 23:25:46.364826 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:46.364800    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1cc0440-3a85-456d-981a-915ae2f4e0bb-kube-api-access-748tb\" (UniqueName: \"kubernetes.io/projected/b1cc0440-3a85-456d-981a-915ae2f4e0bb-kube-api-access-748tb\") pod \"b1cc0440-3a85-456d-981a-915ae2f4e0bb\" (UID: \"b1cc0440-3a85-456d-981a-915ae2f4e0bb\") "
Oct 31 23:25:46.365059 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:46.365028    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1cc0440-3a85-456d-981a-915ae2f4e0bb-model-cache" pod "b1cc0440-3a85-456d-981a-915ae2f4e0bb" (UID: "b1cc0440-3a85-456d-981a-915ae2f4e0bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:46.365137 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:46.365062    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1cc0440-3a85-456d-981a-915ae2f4e0bb-tmp-dir" pod "b1cc0440-3a85-456d-981a-915ae2f4e0bb" (UID: "b1cc0440-3a85-456d-981a-915ae2f4e0bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:46.365243 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:46.365228    4301 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1cc0440-3a85-456d-981a-915ae2f4e0bb-tmp-dir\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:46.365297 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:46.365249    4301 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1cc0440-3a85-456d-981a-915ae2f4e0bb-model-cache\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:46.365388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:46.365342    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1cc0440-3a85-456d-981a-915ae2f4e0bb-home" pod "b1cc0440-3a85-456d-981a-915ae2f4e0bb" (UID: "b1cc0440-3a85-456d-981a-915ae2f4e0bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:46.367367 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:46.367324    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1cc0440-3a85-456d-981a-915ae2f4e0bb-dshm" pod "b1cc0440-3a85-456d-981a-915ae2f4e0bb" (UID: "b1cc0440-3a85-456d-981a-915ae2f4e0bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:25:46.367485 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:46.367414    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1cc0440-3a85-456d-981a-915ae2f4e0bb-tls-certs" pod "b1cc0440-3a85-456d-981a-915ae2f4e0bb" (UID: "b1cc0440-3a85-456d-981a-915ae2f4e0bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:25:46.367569 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:46.367552    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1cc0440-3a85-456d-981a-915ae2f4e0bb-kube-api-access-748tb" pod "b1cc0440-3a85-456d-981a-915ae2f4e0bb" (UID: "b1cc0440-3a85-456d-981a-915ae2f4e0bb"). InnerVolumeSpecName "kube-api-access-748tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:25:46.466842 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:46.466811    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-748tb\" (UniqueName: \"kubernetes.io/projected/b1cc0440-3a85-456d-981a-915ae2f4e0bb-kube-api-access-748tb\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:46.466842 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:46.466841    4301 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1cc0440-3a85-456d-981a-915ae2f4e0bb-tls-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:46.467065 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:46.466855    4301 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1cc0440-3a85-456d-981a-915ae2f4e0bb-dshm\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:46.467065 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:46.466863    4301 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1cc0440-3a85-456d-981a-915ae2f4e0bb-home\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:25:47.231934 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:47.231899    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz" event={"ID":"b1cc0440-3a85-456d-981a-915ae2f4e0bb","Type":"ContainerDied","Data":"3eefd83de4bd3586d9702b06ab663a75003300e56bc51b705b3c6735b3479a23"}
Oct 31 23:25:47.232411 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:47.231945    4301 scope.go:122] "RemoveContainer" containerID="ad356fb1ac3c6dbac69dd222c782823e58db0471c2948d13dd7f17193a8de41b"
Oct 31 23:25:47.232411 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:47.231912    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"
Oct 31 23:25:47.253744 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:47.253718    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"]
Oct 31 23:25:47.263946 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:47.263923    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-255748/simulator-w2-255748-kserve-dff978c5d-hglpz"]
Oct 31 23:25:47.402232 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:47.402200    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" path="/var/lib/kubelet/pods/b1cc0440-3a85-456d-981a-915ae2f4e0bb/volumes"
Oct 31 23:25:49.415789 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:49.415757    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:25:55.320928 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:55.320895    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:25:59.460746 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:59.460651    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"]
Oct 31 23:25:59.461319 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:59.460983    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq" podUID="cdcadd41-cef3-4fbc-9551-ed8dae06f790" containerName="istio-proxy" containerID="cri-o://d1a3317cd014ecdb689a2e97406f15a7370239b8ddf845d0a4eb845546793a86" gracePeriod=30
Oct 31 23:25:59.468095 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:59.463981    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"]
Oct 31 23:25:59.468095 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:59.464190    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722" podUID="ff5f8cb0-0541-4dae-9a86-8c3c0fdda126" containerName="istio-proxy" containerID="cri-o://f8d872a7669672df63aab7645e0c3a3a5e825e5c5c452146ddcda048667201b9" gracePeriod=30
Oct 31 23:25:59.472568 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:59.472543    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"]
Oct 31 23:25:59.473152 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:25:59.473085    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq" podUID="cc182c32-7804-4e3f-b6eb-d22418055d75" containerName="istio-proxy" containerID="cri-o://347cbed007c562620ce4d4e70ee07825b288160e40b24cadcb7d7f352c487873" gracePeriod=30
Oct 31 23:26:02.549700 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:26:02.549661    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:26:02.750554 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:02.750515    4301 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 23:26:02.750729 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:02.750571    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722" podUID="ff5f8cb0-0541-4dae-9a86-8c3c0fdda126" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 23:26:03.905066 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:03.905018    4301 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 23:26:03.905495 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:03.905114    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq" podUID="cdcadd41-cef3-4fbc-9551-ed8dae06f790" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 23:26:05.004964 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:04.999554    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:26:05.127767 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.127734    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-workload-socket\") pod \"cc182c32-7804-4e3f-b6eb-d22418055d75\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") "
Oct 31 23:26:05.127952 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.127797    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-istio-data\") pod \"cc182c32-7804-4e3f-b6eb-d22418055d75\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") "
Oct 31 23:26:05.127952 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.127833    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc182c32-7804-4e3f-b6eb-d22418055d75-istio-token\" (UniqueName: \"kubernetes.io/projected/cc182c32-7804-4e3f-b6eb-d22418055d75-istio-token\") pod \"cc182c32-7804-4e3f-b6eb-d22418055d75\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") "
Oct 31 23:26:05.127952 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.127861    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-workload-certs\") pod \"cc182c32-7804-4e3f-b6eb-d22418055d75\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") "
Oct 31 23:26:05.127952 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.127905    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-istio-envoy\") pod \"cc182c32-7804-4e3f-b6eb-d22418055d75\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") "
Oct 31 23:26:05.127952 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.127948    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc182c32-7804-4e3f-b6eb-d22418055d75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc182c32-7804-4e3f-b6eb-d22418055d75-istiod-ca-cert\") pod \"cc182c32-7804-4e3f-b6eb-d22418055d75\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") "
Oct 31 23:26:05.128225 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.128004    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc182c32-7804-4e3f-b6eb-d22418055d75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc182c32-7804-4e3f-b6eb-d22418055d75-istio-podinfo\") pod \"cc182c32-7804-4e3f-b6eb-d22418055d75\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") "
Oct 31 23:26:05.128225 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.128045    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-credential-socket\") pod \"cc182c32-7804-4e3f-b6eb-d22418055d75\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") "
Oct 31 23:26:05.128225 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.128068    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-workload-socket" pod "cc182c32-7804-4e3f-b6eb-d22418055d75" (UID: "cc182c32-7804-4e3f-b6eb-d22418055d75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:26:05.128225 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.128108    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc182c32-7804-4e3f-b6eb-d22418055d75-kube-api-access-pf4cg\" (UniqueName: \"kubernetes.io/projected/cc182c32-7804-4e3f-b6eb-d22418055d75-kube-api-access-pf4cg\") pod \"cc182c32-7804-4e3f-b6eb-d22418055d75\" (UID: \"cc182c32-7804-4e3f-b6eb-d22418055d75\") "
Oct 31 23:26:05.128520 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.128490    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:26:05.135312 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.135054    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-credential-socket" pod "cc182c32-7804-4e3f-b6eb-d22418055d75" (UID: "cc182c32-7804-4e3f-b6eb-d22418055d75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:26:05.135312 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.135220    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc182c32-7804-4e3f-b6eb-d22418055d75-istiod-ca-cert" pod "cc182c32-7804-4e3f-b6eb-d22418055d75" (UID: "cc182c32-7804-4e3f-b6eb-d22418055d75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:26:05.135312 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.135283    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-workload-certs" pod "cc182c32-7804-4e3f-b6eb-d22418055d75" (UID: "cc182c32-7804-4e3f-b6eb-d22418055d75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:26:05.176711 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.145435    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-istio-data" pod "cc182c32-7804-4e3f-b6eb-d22418055d75" (UID: "cc182c32-7804-4e3f-b6eb-d22418055d75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:26:05.176711 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.148706    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-istio-envoy" pod "cc182c32-7804-4e3f-b6eb-d22418055d75" (UID: "cc182c32-7804-4e3f-b6eb-d22418055d75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:26:05.176711 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.150995    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc182c32-7804-4e3f-b6eb-d22418055d75-istio-podinfo" pod "cc182c32-7804-4e3f-b6eb-d22418055d75" (UID: "cc182c32-7804-4e3f-b6eb-d22418055d75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:26:05.176711 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.152068    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc182c32-7804-4e3f-b6eb-d22418055d75-kube-api-access-pf4cg" pod "cc182c32-7804-4e3f-b6eb-d22418055d75" (UID: "cc182c32-7804-4e3f-b6eb-d22418055d75"). InnerVolumeSpecName "kube-api-access-pf4cg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:26:05.176711 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.154417    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc182c32-7804-4e3f-b6eb-d22418055d75-istio-token" pod "cc182c32-7804-4e3f-b6eb-d22418055d75" (UID: "cc182c32-7804-4e3f-b6eb-d22418055d75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:26:05.225738 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.225712    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:26:05.227808 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.227787    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:26:05.250381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.234607    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-kube-api-access-k59xn\" (UniqueName: \"kubernetes.io/projected/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-kube-api-access-k59xn\") pod \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") "
Oct 31 23:26:05.250381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.234656    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istio-envoy\") pod \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") "
Oct 31 23:26:05.250381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.234689    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-credential-socket\") pod \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") "
Oct 31 23:26:05.250381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.234720    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istio-data\") pod \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") "
Oct 31 23:26:05.250381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.234766    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-workload-certs\") pod \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") "
Oct 31 23:26:05.250381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.234793    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-credential-socket\") pod \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") "
Oct 31 23:26:05.250381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.234826    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdcadd41-cef3-4fbc-9551-ed8dae06f790-kube-api-access-pdxtt\" (UniqueName: \"kubernetes.io/projected/cdcadd41-cef3-4fbc-9551-ed8dae06f790-kube-api-access-pdxtt\") pod \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") "
Oct 31 23:26:05.250381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.234858    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istio-data\") pod \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") "
Oct 31 23:26:05.250381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.234887    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-workload-socket\") pod \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") "
Oct 31 23:26:05.250381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.234921    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-workload-socket\") pod \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") "
Oct 31 23:26:05.250381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.234953    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istiod-ca-cert\") pod \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") "
Oct 31 23:26:05.250381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.235018    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istiod-ca-cert\") pod \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") "
Oct 31 23:26:05.250381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.235047    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istio-podinfo\") pod \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") "
Oct 31 23:26:05.250381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.235091    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-workload-certs\") pod \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") "
Oct 31 23:26:05.250381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.235123    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istio-token\" (UniqueName: \"kubernetes.io/projected/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istio-token\") pod \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") "
Oct 31 23:26:05.250381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.235174    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istio-token\" (UniqueName: \"kubernetes.io/projected/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istio-token\") pod \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") "
Oct 31 23:26:05.250381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.235208    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istio-envoy\") pod \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\" (UID: \"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126\") "
Oct 31 23:26:05.250381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.235294    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istio-podinfo\") pod \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\" (UID: \"cdcadd41-cef3-4fbc-9551-ed8dae06f790\") "
Oct 31 23:26:05.250381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.235668    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc182c32-7804-4e3f-b6eb-d22418055d75-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:26:05.250381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.235690    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:26:05.250381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.235705    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf4cg\" (UniqueName: \"kubernetes.io/projected/cc182c32-7804-4e3f-b6eb-d22418055d75-kube-api-access-pf4cg\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:26:05.250381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.235720    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:26:05.250381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.235736    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc182c32-7804-4e3f-b6eb-d22418055d75-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:26:05.250381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.235750    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:26:05.250381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.235764    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc182c32-7804-4e3f-b6eb-d22418055d75-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:26:05.250381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.235778    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc182c32-7804-4e3f-b6eb-d22418055d75-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:26:05.279602 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.261932    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-credential-socket" pod "cdcadd41-cef3-4fbc-9551-ed8dae06f790" (UID: "cdcadd41-cef3-4fbc-9551-ed8dae06f790"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:26:05.279602 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.262235    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-credential-socket" pod "ff5f8cb0-0541-4dae-9a86-8c3c0fdda126" (UID: "ff5f8cb0-0541-4dae-9a86-8c3c0fdda126"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:26:05.279602 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.275497    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istio-data" pod "ff5f8cb0-0541-4dae-9a86-8c3c0fdda126" (UID: "ff5f8cb0-0541-4dae-9a86-8c3c0fdda126"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:26:05.279602 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.275575    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-workload-certs" pod "cdcadd41-cef3-4fbc-9551-ed8dae06f790" (UID: "cdcadd41-cef3-4fbc-9551-ed8dae06f790"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:26:05.279602 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.279218    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-workload-socket" pod "ff5f8cb0-0541-4dae-9a86-8c3c0fdda126" (UID: "ff5f8cb0-0541-4dae-9a86-8c3c0fdda126"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:26:05.290413 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.281572    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-workload-certs" pod "ff5f8cb0-0541-4dae-9a86-8c3c0fdda126" (UID: "ff5f8cb0-0541-4dae-9a86-8c3c0fdda126"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:26:05.290413 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.282925    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-kube-api-access-k59xn" pod "ff5f8cb0-0541-4dae-9a86-8c3c0fdda126" (UID: "ff5f8cb0-0541-4dae-9a86-8c3c0fdda126"). InnerVolumeSpecName "kube-api-access-k59xn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:26:05.290413 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.283097    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istio-data" pod "cdcadd41-cef3-4fbc-9551-ed8dae06f790" (UID: "cdcadd41-cef3-4fbc-9551-ed8dae06f790"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:26:05.290413 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.283342    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istiod-ca-cert" pod "ff5f8cb0-0541-4dae-9a86-8c3c0fdda126" (UID: "ff5f8cb0-0541-4dae-9a86-8c3c0fdda126"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:26:05.290413 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.283502    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-workload-socket" pod "cdcadd41-cef3-4fbc-9551-ed8dae06f790" (UID: "cdcadd41-cef3-4fbc-9551-ed8dae06f790"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:26:05.290413 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.283722    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istiod-ca-cert" pod "cdcadd41-cef3-4fbc-9551-ed8dae06f790" (UID: "cdcadd41-cef3-4fbc-9551-ed8dae06f790"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:26:05.290413 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.283797    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdcadd41-cef3-4fbc-9551-ed8dae06f790-kube-api-access-pdxtt" pod "cdcadd41-cef3-4fbc-9551-ed8dae06f790" (UID: "cdcadd41-cef3-4fbc-9551-ed8dae06f790"). InnerVolumeSpecName "kube-api-access-pdxtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:26:05.314406 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.312522    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istio-podinfo" pod "ff5f8cb0-0541-4dae-9a86-8c3c0fdda126" (UID: "ff5f8cb0-0541-4dae-9a86-8c3c0fdda126"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:26:05.314406 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.312640    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istio-token" pod "cdcadd41-cef3-4fbc-9551-ed8dae06f790" (UID: "cdcadd41-cef3-4fbc-9551-ed8dae06f790"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:26:05.314406 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.312703    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istio-token" pod "ff5f8cb0-0541-4dae-9a86-8c3c0fdda126" (UID: "ff5f8cb0-0541-4dae-9a86-8c3c0fdda126"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:26:05.314406 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.312762    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istio-envoy" pod "ff5f8cb0-0541-4dae-9a86-8c3c0fdda126" (UID: "ff5f8cb0-0541-4dae-9a86-8c3c0fdda126"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:26:05.315019 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.314960    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istio-podinfo" pod "cdcadd41-cef3-4fbc-9551-ed8dae06f790" (UID: "cdcadd41-cef3-4fbc-9551-ed8dae06f790"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:26:05.315019 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.314973    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istio-envoy" pod "cdcadd41-cef3-4fbc-9551-ed8dae06f790" (UID: "cdcadd41-cef3-4fbc-9551-ed8dae06f790"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:26:05.340108 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.338901    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:26:05.340108 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.338941    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k59xn\" (UniqueName: \"kubernetes.io/projected/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-kube-api-access-k59xn\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:26:05.340108 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.338959    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:26:05.340108 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.338974    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:26:05.340108 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.338990    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:26:05.340108 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.339004    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:26:05.340108 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.339017    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:26:05.340108 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.339031    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdxtt\" (UniqueName: \"kubernetes.io/projected/cdcadd41-cef3-4fbc-9551-ed8dae06f790-kube-api-access-pdxtt\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:26:05.340108 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.339046    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:26:05.340108 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.339059    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:26:05.340108 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.339074    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdcadd41-cef3-4fbc-9551-ed8dae06f790-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:26:05.340108 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.339087    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:26:05.340108 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.339100    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:26:05.340108 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.339114    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:26:05.340108 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.339127    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:26:05.340108 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.339140    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdcadd41-cef3-4fbc-9551-ed8dae06f790-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:26:05.340108 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.339154    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:26:05.340108 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.339167    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:26:05.414958 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.414906    4301 generic.go:356] "Generic (PLEG): container finished" podID="cc182c32-7804-4e3f-b6eb-d22418055d75" containerID="347cbed007c562620ce4d4e70ee07825b288160e40b24cadcb7d7f352c487873" exitCode=0
Oct 31 23:26:05.415138 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.415123    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"
Oct 31 23:26:05.416815 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.416780    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq" event={"ID":"cc182c32-7804-4e3f-b6eb-d22418055d75","Type":"ContainerDied","Data":"347cbed007c562620ce4d4e70ee07825b288160e40b24cadcb7d7f352c487873"}
Oct 31 23:26:05.416957 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.416841    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq" event={"ID":"cc182c32-7804-4e3f-b6eb-d22418055d75","Type":"ContainerDied","Data":"467f2796fba9a78feea970bfd89d5339722815972fa4f941313a98e5736b75a8"}
Oct 31 23:26:05.416957 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.416872    4301 scope.go:122] "RemoveContainer" containerID="347cbed007c562620ce4d4e70ee07825b288160e40b24cadcb7d7f352c487873"
Oct 31 23:26:05.424767 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.419665    4301 generic.go:356] "Generic (PLEG): container finished" podID="ff5f8cb0-0541-4dae-9a86-8c3c0fdda126" containerID="f8d872a7669672df63aab7645e0c3a3a5e825e5c5c452146ddcda048667201b9" exitCode=0
Oct 31 23:26:05.424767 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.419847    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722" event={"ID":"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126","Type":"ContainerDied","Data":"f8d872a7669672df63aab7645e0c3a3a5e825e5c5c452146ddcda048667201b9"}
Oct 31 23:26:05.424767 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.419893    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722" event={"ID":"ff5f8cb0-0541-4dae-9a86-8c3c0fdda126","Type":"ContainerDied","Data":"776e0e0e54c588f184213be813c27164cc619bd01760f1fee79311bd96948e8b"}
Oct 31 23:26:05.424767 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.419981    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"
Oct 31 23:26:05.424767 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.424728    4301 generic.go:356] "Generic (PLEG): container finished" podID="cdcadd41-cef3-4fbc-9551-ed8dae06f790" containerID="d1a3317cd014ecdb689a2e97406f15a7370239b8ddf845d0a4eb845546793a86" exitCode=0
Oct 31 23:26:05.425201 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.424797    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq" event={"ID":"cdcadd41-cef3-4fbc-9551-ed8dae06f790","Type":"ContainerDied","Data":"d1a3317cd014ecdb689a2e97406f15a7370239b8ddf845d0a4eb845546793a86"}
Oct 31 23:26:05.425201 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.424837    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq" event={"ID":"cdcadd41-cef3-4fbc-9551-ed8dae06f790","Type":"ContainerDied","Data":"152b69f50aad27a1f7c30d13631cff355b04e1f958da09ffdda0e9bec758bde7"}
Oct 31 23:26:05.425201 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.424904    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"
Oct 31 23:26:05.510613 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.510582    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"]
Oct 31 23:26:05.517556 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.517475    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-sr9bq"]
Oct 31 23:26:05.584588 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.584484    4301 scope.go:122] "RemoveContainer" containerID="347cbed007c562620ce4d4e70ee07825b288160e40b24cadcb7d7f352c487873"
Oct 31 23:26:05.586518 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:26:05.584943    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"347cbed007c562620ce4d4e70ee07825b288160e40b24cadcb7d7f352c487873\": container with ID starting with 347cbed007c562620ce4d4e70ee07825b288160e40b24cadcb7d7f352c487873 not found: ID does not exist" containerID="347cbed007c562620ce4d4e70ee07825b288160e40b24cadcb7d7f352c487873"
Oct 31 23:26:05.586518 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.584980    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"347cbed007c562620ce4d4e70ee07825b288160e40b24cadcb7d7f352c487873"} err="failed to get container status \"347cbed007c562620ce4d4e70ee07825b288160e40b24cadcb7d7f352c487873\": rpc error: code = NotFound desc = could not find container \"347cbed007c562620ce4d4e70ee07825b288160e40b24cadcb7d7f352c487873\": container with ID starting with 347cbed007c562620ce4d4e70ee07825b288160e40b24cadcb7d7f352c487873 not found: ID does not exist"
Oct 31 23:26:05.586518 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.585002    4301 scope.go:122] "RemoveContainer" containerID="f8d872a7669672df63aab7645e0c3a3a5e825e5c5c452146ddcda048667201b9"
Oct 31 23:26:05.600011 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.599600    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"]
Oct 31 23:26:05.603734 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.603709    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-lkcsq"]
Oct 31 23:26:05.610756 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.610431    4301 scope.go:122] "RemoveContainer" containerID="f8d872a7669672df63aab7645e0c3a3a5e825e5c5c452146ddcda048667201b9"
Oct 31 23:26:05.610911 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:26:05.610882    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8d872a7669672df63aab7645e0c3a3a5e825e5c5c452146ddcda048667201b9\": container with ID starting with f8d872a7669672df63aab7645e0c3a3a5e825e5c5c452146ddcda048667201b9 not found: ID does not exist" containerID="f8d872a7669672df63aab7645e0c3a3a5e825e5c5c452146ddcda048667201b9"
Oct 31 23:26:05.610959 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.610923    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8d872a7669672df63aab7645e0c3a3a5e825e5c5c452146ddcda048667201b9"} err="failed to get container status \"f8d872a7669672df63aab7645e0c3a3a5e825e5c5c452146ddcda048667201b9\": rpc error: code = NotFound desc = could not find container \"f8d872a7669672df63aab7645e0c3a3a5e825e5c5c452146ddcda048667201b9\": container with ID starting with f8d872a7669672df63aab7645e0c3a3a5e825e5c5c452146ddcda048667201b9 not found: ID does not exist"
Oct 31 23:26:05.610959 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.610951    4301 scope.go:122] "RemoveContainer" containerID="d1a3317cd014ecdb689a2e97406f15a7370239b8ddf845d0a4eb845546793a86"
Oct 31 23:26:05.616189 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.616149    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"]
Oct 31 23:26:05.621996 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.621968    4301 scope.go:122] "RemoveContainer" containerID="d1a3317cd014ecdb689a2e97406f15a7370239b8ddf845d0a4eb845546793a86"
Oct 31 23:26:05.622294 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:26:05.622275    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1a3317cd014ecdb689a2e97406f15a7370239b8ddf845d0a4eb845546793a86\": container with ID starting with d1a3317cd014ecdb689a2e97406f15a7370239b8ddf845d0a4eb845546793a86 not found: ID does not exist" containerID="d1a3317cd014ecdb689a2e97406f15a7370239b8ddf845d0a4eb845546793a86"
Oct 31 23:26:05.622396 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.622299    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1a3317cd014ecdb689a2e97406f15a7370239b8ddf845d0a4eb845546793a86"} err="failed to get container status \"d1a3317cd014ecdb689a2e97406f15a7370239b8ddf845d0a4eb845546793a86\": rpc error: code = NotFound desc = could not find container \"d1a3317cd014ecdb689a2e97406f15a7370239b8ddf845d0a4eb845546793a86\": container with ID starting with d1a3317cd014ecdb689a2e97406f15a7370239b8ddf845d0a4eb845546793a86 not found: ID does not exist"
Oct 31 23:26:05.622667 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:05.622648    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w8722"]
Oct 31 23:26:07.402272 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:07.402234    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc182c32-7804-4e3f-b6eb-d22418055d75" path="/var/lib/kubelet/pods/cc182c32-7804-4e3f-b6eb-d22418055d75/volumes"
Oct 31 23:26:07.402860 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:07.402840    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdcadd41-cef3-4fbc-9551-ed8dae06f790" path="/var/lib/kubelet/pods/cdcadd41-cef3-4fbc-9551-ed8dae06f790/volumes"
Oct 31 23:26:07.403291 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:07.403272    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff5f8cb0-0541-4dae-9a86-8c3c0fdda126" path="/var/lib/kubelet/pods/ff5f8cb0-0541-4dae-9a86-8c3c0fdda126/volumes"
Oct 31 23:26:32.545324 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:26:32.545238    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:26:42.882560 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:26:42.882527    4301 scope.go:122] "RemoveContainer" containerID="d4cca9d627153fe04dd17cc11e7b3165096b3c36e35ea945cf0ec7a47fb5055f"
Oct 31 23:27:02.544553 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:27:02.544510    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:27:08.294967 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:27:08.294931    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:27:20.021414 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:27:20.021379    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94ecade3-w8zp7"]
Oct 31 23:27:20.024303 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:27:20.024273    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94ecade3-w8zp7"]
Oct 31 23:27:21.407281 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:27:21.407246    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05c2fd23-331c-4818-b14c-786d6e14477a" path="/var/lib/kubelet/pods/05c2fd23-331c-4818-b14c-786d6e14477a/volumes"
Oct 31 23:27:25.027488 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:27:25.027445    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e595be3-jrjfx"]
Oct 31 23:27:25.029780 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:27:25.029759    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e595be3-jrjfx"]
Oct 31 23:27:25.403078 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:27:25.403042    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53842309-02b2-4ce4-a9e6-334334c57b84" path="/var/lib/kubelet/pods/53842309-02b2-4ce4-a9e6-334334c57b84/volumes"
Oct 31 23:27:26.394111 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:27:26.394074    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:27:30.675383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:27:30.674742    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:27:32.545015 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:27:32.544976    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:27:42.918241 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:27:42.918208    4301 scope.go:122] "RemoveContainer" containerID="e8f35cbd467eb603ddcac5b5c41fa555104ac1c32843b6a1d6ccd98a242d8c06"
Oct 31 23:27:45.384016 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:27:45.383981    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:27:58.219391 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:27:58.219299    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:28:02.544842 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:28:02.544805    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:28:23.095104 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:28:23.095072    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:28:31.021583 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:28:31.021546    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1faf30d6-dvscm"]
Oct 31 23:28:31.025101 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:28:31.025070    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1faf30d6-dvscm"]
Oct 31 23:28:31.403113 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:28:31.403084    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b05705b9-6c2f-4c7f-9d58-e832c56afde5" path="/var/lib/kubelet/pods/b05705b9-6c2f-4c7f-9d58-e832c56afde5/volumes"
Oct 31 23:28:32.547895 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:28:32.547857    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:28:33.392668 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:28:33.392638    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:28:37.213899 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:28:37.213861    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:28:42.938097 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:28:42.938064    4301 scope.go:122] "RemoveContainer" containerID="252166cfcd8cbb9a1615a6065a46d543422b1bad7ee2a56d7fd60ba7b592d104"
Oct 31 23:28:42.946442 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:28:42.946416    4301 scope.go:122] "RemoveContainer" containerID="2cd5f93792ccafcd8dc80c5a888176640cacecd5b4afb878b9c466d28662e0da"
Oct 31 23:28:42.953759 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:28:42.953741    4301 scope.go:122] "RemoveContainer" containerID="72fe69afc94e8b31ecbc159e8d5d7bb18793b87acd3fb4583c46aa95172cb073"
Oct 31 23:28:56.188892 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:28:56.188855    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:29:00.372469 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:00.372411    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:29:02.544957 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:29:02.544921    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:29:14.488481 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:14.488442    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:29:20.357547 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:20.357509    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:29:32.545477 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:29:32.545384    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:29:33.091000 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:33.090966    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:29:36.987382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:36.986514    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:29:41.826119 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:41.826086    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s9nl_1f0042ee-6258-4aa8-af48-7568f2f2e8a1/ovn-acl-logging/0.log"
Oct 31 23:29:41.826563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:41.826447    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s9nl_1f0042ee-6258-4aa8-af48-7568f2f2e8a1/ovn-acl-logging/0.log"
Oct 31 23:29:42.977934 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:42.977903    4301 scope.go:122] "RemoveContainer" containerID="1f7e3b473785cb1acd986fdb37437e78cffb5afca5ab705abf121c842fcee035"
Oct 31 23:29:42.986188 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:42.986171    4301 scope.go:122] "RemoveContainer" containerID="b0f29c1197dd0cf4d77eaaf49248ade2e2dc29fb32b2b103daa885a3b8b527b2"
Oct 31 23:29:42.993761 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:42.993741    4301 scope.go:122] "RemoveContainer" containerID="6f5795b70347257a963100a1f7ab094e8f4a12287a4938159d12cce0364e68db"
Oct 31 23:29:43.001120 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:43.001099    4301 scope.go:122] "RemoveContainer" containerID="31c5f1148655812fc527f4593d8a06e1a69ff0768487851fead68f144e676e97"
Oct 31 23:29:43.008231 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:43.008197    4301 scope.go:122] "RemoveContainer" containerID="5c40bb0b084ae520954888258cc6f2d5d268c67a36d31741d38aa0004ae1ce0c"
Oct 31 23:29:43.015216 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:43.015200    4301 scope.go:122] "RemoveContainer" containerID="286b3fb755963c623ecffb17c1e45670fdac2bee61228b02e4d1f85eb9087478"
Oct 31 23:29:48.290061 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:48.290027    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b"]
Oct 31 23:29:48.290556 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:48.290235    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b" podUID="b331f886-ca34-4c4b-a31c-d2e45aac6af3" containerName="discovery" containerID="cri-o://ac276961c29947a2357957adc6b3d71e34bece94120ed88885ba0df52407edc9" gracePeriod=30
Oct 31 23:29:48.540653 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:48.540598    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b"
Oct 31 23:29:48.652937 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:48.652898    4301 generic.go:356] "Generic (PLEG): container finished" podID="b331f886-ca34-4c4b-a31c-d2e45aac6af3" containerID="ac276961c29947a2357957adc6b3d71e34bece94120ed88885ba0df52407edc9" exitCode=0
Oct 31 23:29:48.653110 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:48.652952    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b"
Oct 31 23:29:48.653110 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:48.652978    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b" event={"ID":"b331f886-ca34-4c4b-a31c-d2e45aac6af3","Type":"ContainerDied","Data":"ac276961c29947a2357957adc6b3d71e34bece94120ed88885ba0df52407edc9"}
Oct 31 23:29:48.653110 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:48.653030    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b" event={"ID":"b331f886-ca34-4c4b-a31c-d2e45aac6af3","Type":"ContainerDied","Data":"0ee0dfafce6f405cc36f5d3ae3f3d1f22bb11c7eb9070b03900e58d01335014f"}
Oct 31 23:29:48.653110 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:48.653049    4301 scope.go:122] "RemoveContainer" containerID="ac276961c29947a2357957adc6b3d71e34bece94120ed88885ba0df52407edc9"
Oct 31 23:29:48.662060 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:48.662042    4301 scope.go:122] "RemoveContainer" containerID="ac276961c29947a2357957adc6b3d71e34bece94120ed88885ba0df52407edc9"
Oct 31 23:29:48.662329 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:29:48.662311    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac276961c29947a2357957adc6b3d71e34bece94120ed88885ba0df52407edc9\": container with ID starting with ac276961c29947a2357957adc6b3d71e34bece94120ed88885ba0df52407edc9 not found: ID does not exist" containerID="ac276961c29947a2357957adc6b3d71e34bece94120ed88885ba0df52407edc9"
Oct 31 23:29:48.662400 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:48.662340    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac276961c29947a2357957adc6b3d71e34bece94120ed88885ba0df52407edc9"} err="failed to get container status \"ac276961c29947a2357957adc6b3d71e34bece94120ed88885ba0df52407edc9\": rpc error: code = NotFound desc = could not find container \"ac276961c29947a2357957adc6b3d71e34bece94120ed88885ba0df52407edc9\": container with ID starting with ac276961c29947a2357957adc6b3d71e34bece94120ed88885ba0df52407edc9 not found: ID does not exist"
Oct 31 23:29:48.692420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:48.692347    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b331f886-ca34-4c4b-a31c-d2e45aac6af3-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b331f886-ca34-4c4b-a31c-d2e45aac6af3-local-certs\") pod \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\" (UID: \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\") "
Oct 31 23:29:48.692575 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:48.692467    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b331f886-ca34-4c4b-a31c-d2e45aac6af3-cacerts\" (UniqueName: \"kubernetes.io/secret/b331f886-ca34-4c4b-a31c-d2e45aac6af3-cacerts\") pod \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\" (UID: \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\") "
Oct 31 23:29:48.692575 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:48.692497    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b331f886-ca34-4c4b-a31c-d2e45aac6af3-kube-api-access-fvfzz\" (UniqueName: \"kubernetes.io/projected/b331f886-ca34-4c4b-a31c-d2e45aac6af3-kube-api-access-fvfzz\") pod \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\" (UID: \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\") "
Oct 31 23:29:48.692575 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:48.692517    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b331f886-ca34-4c4b-a31c-d2e45aac6af3-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b331f886-ca34-4c4b-a31c-d2e45aac6af3-istio-kubeconfig\") pod \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\" (UID: \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\") "
Oct 31 23:29:48.692740 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:48.692681    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b331f886-ca34-4c4b-a31c-d2e45aac6af3-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b331f886-ca34-4c4b-a31c-d2e45aac6af3-istio-csr-dns-cert\") pod \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\" (UID: \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\") "
Oct 31 23:29:48.692799 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:48.692745    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b331f886-ca34-4c4b-a31c-d2e45aac6af3-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b331f886-ca34-4c4b-a31c-d2e45aac6af3-istio-csr-ca-configmap\") pod \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\" (UID: \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\") "
Oct 31 23:29:48.692799 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:48.692783    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b331f886-ca34-4c4b-a31c-d2e45aac6af3-istio-token\" (UniqueName: \"kubernetes.io/projected/b331f886-ca34-4c4b-a31c-d2e45aac6af3-istio-token\") pod \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\" (UID: \"b331f886-ca34-4c4b-a31c-d2e45aac6af3\") "
Oct 31 23:29:48.693166 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:48.693135    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b331f886-ca34-4c4b-a31c-d2e45aac6af3-istio-csr-ca-configmap" pod "b331f886-ca34-4c4b-a31c-d2e45aac6af3" (UID: "b331f886-ca34-4c4b-a31c-d2e45aac6af3"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:29:48.694767 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:48.694740    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b331f886-ca34-4c4b-a31c-d2e45aac6af3-istio-kubeconfig" pod "b331f886-ca34-4c4b-a31c-d2e45aac6af3" (UID: "b331f886-ca34-4c4b-a31c-d2e45aac6af3"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:29:48.694892 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:48.694777    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b331f886-ca34-4c4b-a31c-d2e45aac6af3-cacerts" pod "b331f886-ca34-4c4b-a31c-d2e45aac6af3" (UID: "b331f886-ca34-4c4b-a31c-d2e45aac6af3"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:29:48.694956 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:48.694893    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b331f886-ca34-4c4b-a31c-d2e45aac6af3-local-certs" pod "b331f886-ca34-4c4b-a31c-d2e45aac6af3" (UID: "b331f886-ca34-4c4b-a31c-d2e45aac6af3"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:29:48.695020 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:48.695003    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b331f886-ca34-4c4b-a31c-d2e45aac6af3-kube-api-access-fvfzz" pod "b331f886-ca34-4c4b-a31c-d2e45aac6af3" (UID: "b331f886-ca34-4c4b-a31c-d2e45aac6af3"). InnerVolumeSpecName "kube-api-access-fvfzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:29:48.695258 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:48.695237    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b331f886-ca34-4c4b-a31c-d2e45aac6af3-istio-csr-dns-cert" pod "b331f886-ca34-4c4b-a31c-d2e45aac6af3" (UID: "b331f886-ca34-4c4b-a31c-d2e45aac6af3"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:29:48.695321 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:48.695234    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b331f886-ca34-4c4b-a31c-d2e45aac6af3-istio-token" pod "b331f886-ca34-4c4b-a31c-d2e45aac6af3" (UID: "b331f886-ca34-4c4b-a31c-d2e45aac6af3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:29:48.794017 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:48.793938    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvfzz\" (UniqueName: \"kubernetes.io/projected/b331f886-ca34-4c4b-a31c-d2e45aac6af3-kube-api-access-fvfzz\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:29:48.794017 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:48.793963    4301 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b331f886-ca34-4c4b-a31c-d2e45aac6af3-istio-kubeconfig\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:29:48.794017 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:48.793972    4301 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b331f886-ca34-4c4b-a31c-d2e45aac6af3-istio-csr-dns-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:29:48.794017 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:48.793984    4301 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b331f886-ca34-4c4b-a31c-d2e45aac6af3-istio-csr-ca-configmap\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:29:48.794017 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:48.793997    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b331f886-ca34-4c4b-a31c-d2e45aac6af3-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:29:48.794017 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:48.794005    4301 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b331f886-ca34-4c4b-a31c-d2e45aac6af3-local-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:29:48.794017 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:48.794013    4301 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b331f886-ca34-4c4b-a31c-d2e45aac6af3-cacerts\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:29:48.974246 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:48.974217    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b"]
Oct 31 23:29:48.978492 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:48.978466    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pvl9b"]
Oct 31 23:29:49.027789 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:49.027755    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de6d2120-x6jhx"]
Oct 31 23:29:49.031074 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:49.031052    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de6d2120-x6jhx"]
Oct 31 23:29:49.402013 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:49.401982    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9760fd20-c563-4f5a-be5a-eeca1a7898a3" path="/var/lib/kubelet/pods/9760fd20-c563-4f5a-be5a-eeca1a7898a3/volumes"
Oct 31 23:29:49.402441 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:49.402427    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b331f886-ca34-4c4b-a31c-d2e45aac6af3" path="/var/lib/kubelet/pods/b331f886-ca34-4c4b-a31c-d2e45aac6af3/volumes"
Oct 31 23:29:50.917622 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:29:50.917571    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:30:01.703197 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:01.703160    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:30:02.550911 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:30:02.550866    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:30:05.022014 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:05.021968    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f98a428-bzspr"]
Oct 31 23:30:05.024957 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:05.024933    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f98a428-bzspr"]
Oct 31 23:30:05.404655 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:05.404611    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fbc8cf6-9cce-414c-9146-8a2ebd840fe0" path="/var/lib/kubelet/pods/2fbc8cf6-9cce-414c-9146-8a2ebd840fe0/volumes"
Oct 31 23:30:07.014229 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:07.014194    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:30:23.014329 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:23.014294    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:30:26.284654 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:26.284615    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:30:32.545894 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:30:32.545856    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:30:33.430804 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:33.430767    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:30:43.049438 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:43.049405    4301 scope.go:122] "RemoveContainer" containerID="b49bd1b04c356737e52b6b8109a85a709ad6f60cd47b09da1fc5ab81f8b444b4"
Oct 31 23:30:48.411624 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:48.411589    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g"]
Oct 31 23:30:48.412195 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:48.411782    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g" podUID="396eb171-d982-4c65-8ce4-0726cb0e29e5" containerName="discovery" containerID="cri-o://c195a186a6db22b8b8955fd82bfb2d12f616a68c07003ccf851a19073df4bfa6" gracePeriod=30
Oct 31 23:30:48.661144 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:48.661119    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g"
Oct 31 23:30:48.711002 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:48.710912    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/396eb171-d982-4c65-8ce4-0726cb0e29e5-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/396eb171-d982-4c65-8ce4-0726cb0e29e5-istio-kubeconfig\") pod \"396eb171-d982-4c65-8ce4-0726cb0e29e5\" (UID: \"396eb171-d982-4c65-8ce4-0726cb0e29e5\") "
Oct 31 23:30:48.711002 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:48.710969    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/396eb171-d982-4c65-8ce4-0726cb0e29e5-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/396eb171-d982-4c65-8ce4-0726cb0e29e5-istio-csr-ca-configmap\") pod \"396eb171-d982-4c65-8ce4-0726cb0e29e5\" (UID: \"396eb171-d982-4c65-8ce4-0726cb0e29e5\") "
Oct 31 23:30:48.711242 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:48.711006    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/396eb171-d982-4c65-8ce4-0726cb0e29e5-istio-token\" (UniqueName: \"kubernetes.io/projected/396eb171-d982-4c65-8ce4-0726cb0e29e5-istio-token\") pod \"396eb171-d982-4c65-8ce4-0726cb0e29e5\" (UID: \"396eb171-d982-4c65-8ce4-0726cb0e29e5\") "
Oct 31 23:30:48.711242 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:48.711059    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/396eb171-d982-4c65-8ce4-0726cb0e29e5-cacerts\" (UniqueName: \"kubernetes.io/secret/396eb171-d982-4c65-8ce4-0726cb0e29e5-cacerts\") pod \"396eb171-d982-4c65-8ce4-0726cb0e29e5\" (UID: \"396eb171-d982-4c65-8ce4-0726cb0e29e5\") "
Oct 31 23:30:48.711242 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:48.711099    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/396eb171-d982-4c65-8ce4-0726cb0e29e5-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/396eb171-d982-4c65-8ce4-0726cb0e29e5-local-certs\") pod \"396eb171-d982-4c65-8ce4-0726cb0e29e5\" (UID: \"396eb171-d982-4c65-8ce4-0726cb0e29e5\") "
Oct 31 23:30:48.711242 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:48.711163    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/396eb171-d982-4c65-8ce4-0726cb0e29e5-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/396eb171-d982-4c65-8ce4-0726cb0e29e5-istio-csr-dns-cert\") pod \"396eb171-d982-4c65-8ce4-0726cb0e29e5\" (UID: \"396eb171-d982-4c65-8ce4-0726cb0e29e5\") "
Oct 31 23:30:48.711242 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:48.711192    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/396eb171-d982-4c65-8ce4-0726cb0e29e5-kube-api-access-jrrqq\" (UniqueName: \"kubernetes.io/projected/396eb171-d982-4c65-8ce4-0726cb0e29e5-kube-api-access-jrrqq\") pod \"396eb171-d982-4c65-8ce4-0726cb0e29e5\" (UID: \"396eb171-d982-4c65-8ce4-0726cb0e29e5\") "
Oct 31 23:30:48.711529 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:48.711420    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/396eb171-d982-4c65-8ce4-0726cb0e29e5-istio-csr-ca-configmap" pod "396eb171-d982-4c65-8ce4-0726cb0e29e5" (UID: "396eb171-d982-4c65-8ce4-0726cb0e29e5"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:30:48.711529 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:48.711505    4301 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/396eb171-d982-4c65-8ce4-0726cb0e29e5-istio-csr-ca-configmap\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:30:48.713875 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:48.713838    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/396eb171-d982-4c65-8ce4-0726cb0e29e5-cacerts" pod "396eb171-d982-4c65-8ce4-0726cb0e29e5" (UID: "396eb171-d982-4c65-8ce4-0726cb0e29e5"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:30:48.714008 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:48.713955    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/396eb171-d982-4c65-8ce4-0726cb0e29e5-istio-csr-dns-cert" pod "396eb171-d982-4c65-8ce4-0726cb0e29e5" (UID: "396eb171-d982-4c65-8ce4-0726cb0e29e5"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:30:48.714070 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:48.714053    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/396eb171-d982-4c65-8ce4-0726cb0e29e5-istio-kubeconfig" pod "396eb171-d982-4c65-8ce4-0726cb0e29e5" (UID: "396eb171-d982-4c65-8ce4-0726cb0e29e5"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:30:48.714163 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:48.714139    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/396eb171-d982-4c65-8ce4-0726cb0e29e5-istio-token" pod "396eb171-d982-4c65-8ce4-0726cb0e29e5" (UID: "396eb171-d982-4c65-8ce4-0726cb0e29e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:30:48.714163 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:48.714151    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/396eb171-d982-4c65-8ce4-0726cb0e29e5-kube-api-access-jrrqq" pod "396eb171-d982-4c65-8ce4-0726cb0e29e5" (UID: "396eb171-d982-4c65-8ce4-0726cb0e29e5"). InnerVolumeSpecName "kube-api-access-jrrqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:30:48.714428 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:48.714401    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/396eb171-d982-4c65-8ce4-0726cb0e29e5-local-certs" pod "396eb171-d982-4c65-8ce4-0726cb0e29e5" (UID: "396eb171-d982-4c65-8ce4-0726cb0e29e5"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:30:48.812256 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:48.812219    4301 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/396eb171-d982-4c65-8ce4-0726cb0e29e5-istio-kubeconfig\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:30:48.812256 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:48.812249    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/396eb171-d982-4c65-8ce4-0726cb0e29e5-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:30:48.812256 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:48.812263    4301 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/396eb171-d982-4c65-8ce4-0726cb0e29e5-cacerts\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:30:48.812560 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:48.812274    4301 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/396eb171-d982-4c65-8ce4-0726cb0e29e5-local-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:30:48.812560 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:48.812288    4301 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/396eb171-d982-4c65-8ce4-0726cb0e29e5-istio-csr-dns-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:30:48.812560 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:48.812301    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrrqq\" (UniqueName: \"kubernetes.io/projected/396eb171-d982-4c65-8ce4-0726cb0e29e5-kube-api-access-jrrqq\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:30:48.941294 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:48.941258    4301 generic.go:356] "Generic (PLEG): container finished" podID="396eb171-d982-4c65-8ce4-0726cb0e29e5" containerID="c195a186a6db22b8b8955fd82bfb2d12f616a68c07003ccf851a19073df4bfa6" exitCode=0
Oct 31 23:30:48.941480 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:48.941307    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g"
Oct 31 23:30:48.941480 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:48.941345    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g" event={"ID":"396eb171-d982-4c65-8ce4-0726cb0e29e5","Type":"ContainerDied","Data":"c195a186a6db22b8b8955fd82bfb2d12f616a68c07003ccf851a19073df4bfa6"}
Oct 31 23:30:48.941480 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:48.941408    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g" event={"ID":"396eb171-d982-4c65-8ce4-0726cb0e29e5","Type":"ContainerDied","Data":"72ca877f0c298e67aed2a4a1fa44f780f0133f285ede175c02b58daff7f84ab8"}
Oct 31 23:30:48.941480 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:48.941432    4301 scope.go:122] "RemoveContainer" containerID="c195a186a6db22b8b8955fd82bfb2d12f616a68c07003ccf851a19073df4bfa6"
Oct 31 23:30:48.958238 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:48.958220    4301 scope.go:122] "RemoveContainer" containerID="c195a186a6db22b8b8955fd82bfb2d12f616a68c07003ccf851a19073df4bfa6"
Oct 31 23:30:48.958504 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:30:48.958486    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c195a186a6db22b8b8955fd82bfb2d12f616a68c07003ccf851a19073df4bfa6\": container with ID starting with c195a186a6db22b8b8955fd82bfb2d12f616a68c07003ccf851a19073df4bfa6 not found: ID does not exist" containerID="c195a186a6db22b8b8955fd82bfb2d12f616a68c07003ccf851a19073df4bfa6"
Oct 31 23:30:48.958561 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:48.958516    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c195a186a6db22b8b8955fd82bfb2d12f616a68c07003ccf851a19073df4bfa6"} err="failed to get container status \"c195a186a6db22b8b8955fd82bfb2d12f616a68c07003ccf851a19073df4bfa6\": rpc error: code = NotFound desc = could not find container \"c195a186a6db22b8b8955fd82bfb2d12f616a68c07003ccf851a19073df4bfa6\": container with ID starting with c195a186a6db22b8b8955fd82bfb2d12f616a68c07003ccf851a19073df4bfa6 not found: ID does not exist"
Oct 31 23:30:48.962861 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:48.962812    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g"]
Oct 31 23:30:48.967004 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:48.966984    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dd96g"]
Oct 31 23:30:49.407210 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:49.407176    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="396eb171-d982-4c65-8ce4-0726cb0e29e5" path="/var/lib/kubelet/pods/396eb171-d982-4c65-8ce4-0726cb0e29e5/volumes"
Oct 31 23:30:51.919022 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:30:51.918989    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:31:02.124382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:31:02.124262    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:31:02.547618 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:31:02.547524    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:31:32.544530 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:31:32.544490    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:31:43.077149 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:31:43.077116    4301 scope.go:122] "RemoveContainer" containerID="8446f14bd9720a8a26f59e60cf422612fabed9660df53d97854b34a4f614203b"
Oct 31 23:31:43.085078 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:31:43.085057    4301 scope.go:122] "RemoveContainer" containerID="a79446064d779a1cf4e8f95ea9a55920862631b3776a212e2cde8c78baa1fb78"
Oct 31 23:32:02.545059 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:32:02.545012    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:32:32.544447 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:32:32.544330    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:33:02.543941 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:33:02.543899    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:33:32.545022 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:33:32.544984    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:34:02.545203 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:34:02.545118    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:34:32.544985 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:34:32.544945    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:34:41.852941 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:34:41.852913    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s9nl_1f0042ee-6258-4aa8-af48-7568f2f2e8a1/ovn-acl-logging/0.log"
Oct 31 23:34:41.854062 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:34:41.854038    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s9nl_1f0042ee-6258-4aa8-af48-7568f2f2e8a1/ovn-acl-logging/0.log"
Oct 31 23:35:02.545171 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:35:02.545127    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:35:32.545621 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:35:32.545538    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:36:02.545246 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:36:02.545211    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:36:20.831803 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:20.831768    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:36:25.101833 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:25.101798    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:36:32.545629 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:36:32.545581    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:36:33.350583 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.350548    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4619f-77c85qzrml"]
Oct 31 23:36:33.350906 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.350891    4301 cpu_manager.go:414] "containerMap: removing container" podUID="cdcadd41-cef3-4fbc-9551-ed8dae06f790" containerName="istio-proxy"
Oct 31 23:36:33.350952 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.350909    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdcadd41-cef3-4fbc-9551-ed8dae06f790" containerName="istio-proxy"
Oct 31 23:36:33.350952 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.350922    4301 cpu_manager.go:414] "containerMap: removing container" podUID="195909b1-e0bf-4679-84e3-50786cc7eaea" containerName="istio-proxy"
Oct 31 23:36:33.350952 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.350927    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="195909b1-e0bf-4679-84e3-50786cc7eaea" containerName="istio-proxy"
Oct 31 23:36:33.350952 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.350933    4301 cpu_manager.go:414] "containerMap: removing container" podUID="d0195c12-01bd-4f4a-a72a-b608150516f2" containerName="istio-proxy"
Oct 31 23:36:33.350952 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.350938    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0195c12-01bd-4f4a-a72a-b608150516f2" containerName="istio-proxy"
Oct 31 23:36:33.350952 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.350948    4301 cpu_manager.go:414] "containerMap: removing container" podUID="d0195c12-01bd-4f4a-a72a-b608150516f2" containerName="istio-proxy"
Oct 31 23:36:33.350952 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.350953    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0195c12-01bd-4f4a-a72a-b608150516f2" containerName="istio-proxy"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.350960    4301 cpu_manager.go:414] "containerMap: removing container" podUID="d361d757-5e4e-43a4-acb7-9ef5f24113af" containerName="istio-proxy"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.350965    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d361d757-5e4e-43a4-acb7-9ef5f24113af" containerName="istio-proxy"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.350974    4301 cpu_manager.go:414] "containerMap: removing container" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.350982    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.350993    4301 cpu_manager.go:414] "containerMap: removing container" podUID="2fbc8cf6-9cce-414c-9146-8a2ebd840fe0" containerName="revoke-keys"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.350997    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fbc8cf6-9cce-414c-9146-8a2ebd840fe0" containerName="revoke-keys"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351003    4301 cpu_manager.go:414] "containerMap: removing container" podUID="cc182c32-7804-4e3f-b6eb-d22418055d75" containerName="istio-proxy"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351007    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc182c32-7804-4e3f-b6eb-d22418055d75" containerName="istio-proxy"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351014    4301 cpu_manager.go:414] "containerMap: removing container" podUID="85e9dd14-aa39-400e-b672-4c3122ea568a" containerName="istio-proxy"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351018    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85e9dd14-aa39-400e-b672-4c3122ea568a" containerName="istio-proxy"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351028    4301 cpu_manager.go:414] "containerMap: removing container" podUID="ff5f8cb0-0541-4dae-9a86-8c3c0fdda126" containerName="istio-proxy"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351032    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff5f8cb0-0541-4dae-9a86-8c3c0fdda126" containerName="istio-proxy"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351037    4301 cpu_manager.go:414] "containerMap: removing container" podUID="f7b097bb-1a53-48cc-8499-f13014278d10" containerName="istio-proxy"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351042    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b097bb-1a53-48cc-8499-f13014278d10" containerName="istio-proxy"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351050    4301 cpu_manager.go:414] "containerMap: removing container" podUID="b331f886-ca34-4c4b-a31c-d2e45aac6af3" containerName="discovery"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351058    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b331f886-ca34-4c4b-a31c-d2e45aac6af3" containerName="discovery"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351067    4301 cpu_manager.go:414] "containerMap: removing container" podUID="92c87d0a-e7ea-44ed-a02d-c92554d9f41e" containerName="istio-proxy"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351072    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92c87d0a-e7ea-44ed-a02d-c92554d9f41e" containerName="istio-proxy"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351082    4301 cpu_manager.go:414] "containerMap: removing container" podUID="a3bc4bb3-e967-4fe3-a6b9-71377f8f6047" containerName="payload-pre-processing"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351086    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3bc4bb3-e967-4fe3-a6b9-71377f8f6047" containerName="payload-pre-processing"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351093    4301 cpu_manager.go:414] "containerMap: removing container" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351097    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351107    4301 cpu_manager.go:414] "containerMap: removing container" podUID="fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5" containerName="istio-proxy"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351111    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5" containerName="istio-proxy"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351119    4301 cpu_manager.go:414] "containerMap: removing container" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351123    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351131    4301 cpu_manager.go:414] "containerMap: removing container" podUID="2abed568-6cc8-4336-b427-b8a976686c3f" containerName="istio-proxy"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351138    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2abed568-6cc8-4336-b427-b8a976686c3f" containerName="istio-proxy"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351147    4301 cpu_manager.go:414] "containerMap: removing container" podUID="bf839f5e-2570-44bd-9c62-7a65ea1590ed" containerName="istio-proxy"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351152    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf839f5e-2570-44bd-9c62-7a65ea1590ed" containerName="istio-proxy"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351158    4301 cpu_manager.go:414] "containerMap: removing container" podUID="396eb171-d982-4c65-8ce4-0726cb0e29e5" containerName="discovery"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351162    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="396eb171-d982-4c65-8ce4-0726cb0e29e5" containerName="discovery"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351168    4301 cpu_manager.go:414] "containerMap: removing container" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351172    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351180    4301 cpu_manager.go:414] "containerMap: removing container" podUID="7a96c54a-c66d-466f-b2d4-84a208c3c664" containerName="istio-proxy"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351184    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a96c54a-c66d-466f-b2d4-84a208c3c664" containerName="istio-proxy"
Oct 31 23:36:33.351173 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351190    4301 cpu_manager.go:414] "containerMap: removing container" podUID="9aac5700-d634-407f-98f1-e7ab6beeebc8" containerName="maas-api"
Oct 31 23:36:33.352298 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351195    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aac5700-d634-407f-98f1-e7ab6beeebc8" containerName="maas-api"
Oct 31 23:36:33.352298 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351248    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aac5700-d634-407f-98f1-e7ab6beeebc8" containerName="maas-api"
Oct 31 23:36:33.352298 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351255    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdcadd41-cef3-4fbc-9551-ed8dae06f790" containerName="istio-proxy"
Oct 31 23:36:33.352298 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351260    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="195909b1-e0bf-4679-84e3-50786cc7eaea" containerName="istio-proxy"
Oct 31 23:36:33.352298 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351266    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0195c12-01bd-4f4a-a72a-b608150516f2" containerName="istio-proxy"
Oct 31 23:36:33.352298 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351273    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a58ab83-2394-48a8-aa4f-04c22ee67a24" containerName="istio-proxy"
Oct 31 23:36:33.352298 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351295    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0195c12-01bd-4f4a-a72a-b608150516f2" containerName="istio-proxy"
Oct 31 23:36:33.352298 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351301    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="d361d757-5e4e-43a4-acb7-9ef5f24113af" containerName="istio-proxy"
Oct 31 23:36:33.352298 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351306    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1cc0440-3a85-456d-981a-915ae2f4e0bb" containerName="main"
Oct 31 23:36:33.352298 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351311    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fbc8cf6-9cce-414c-9146-8a2ebd840fe0" containerName="revoke-keys"
Oct 31 23:36:33.352298 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351317    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc182c32-7804-4e3f-b6eb-d22418055d75" containerName="istio-proxy"
Oct 31 23:36:33.352298 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351322    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="85e9dd14-aa39-400e-b672-4c3122ea568a" containerName="istio-proxy"
Oct 31 23:36:33.352298 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351328    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff5f8cb0-0541-4dae-9a86-8c3c0fdda126" containerName="istio-proxy"
Oct 31 23:36:33.352298 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351333    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7b097bb-1a53-48cc-8499-f13014278d10" containerName="istio-proxy"
Oct 31 23:36:33.352298 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351338    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="b331f886-ca34-4c4b-a31c-d2e45aac6af3" containerName="discovery"
Oct 31 23:36:33.352298 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351342    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="92c87d0a-e7ea-44ed-a02d-c92554d9f41e" containerName="istio-proxy"
Oct 31 23:36:33.352298 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351348    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3bc4bb3-e967-4fe3-a6b9-71377f8f6047" containerName="payload-pre-processing"
Oct 31 23:36:33.352298 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351353    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="570b97e3-1e23-474c-beb7-9438692c5508" containerName="main"
Oct 31 23:36:33.352298 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351385    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc3fa555-8bcd-49d7-ac81-18cd8ddf8ab5" containerName="istio-proxy"
Oct 31 23:36:33.352298 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351394    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9a165f7-116b-4c15-9836-349bebeab135" containerName="main"
Oct 31 23:36:33.352298 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351400    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="2abed568-6cc8-4336-b427-b8a976686c3f" containerName="istio-proxy"
Oct 31 23:36:33.352298 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351405    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf839f5e-2570-44bd-9c62-7a65ea1590ed" containerName="istio-proxy"
Oct 31 23:36:33.352298 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351410    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="396eb171-d982-4c65-8ce4-0726cb0e29e5" containerName="discovery"
Oct 31 23:36:33.352298 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351415    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="1486aeca-b15c-4611-beda-0baf2e503a89" containerName="main"
Oct 31 23:36:33.352298 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.351419    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a96c54a-c66d-466f-b2d4-84a208c3c664" containerName="istio-proxy"
Oct 31 23:36:33.354401 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.354384    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4619f-77c85qzrml"
Oct 31 23:36:33.358238 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.358212    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-957f7454-dockercfg-gqww7\""
Oct 31 23:36:33.359710 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.359687    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-c4619f\""
Oct 31 23:36:33.364323 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.364296    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4619f-77c85qzrml"]
Oct 31 23:36:33.387655 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.387622    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c4619f-6dbc6c67dp8772"]
Oct 31 23:36:33.387975 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.387961    4301 cpu_manager.go:414] "containerMap: removing container" podUID="195909b1-e0bf-4679-84e3-50786cc7eaea" containerName="istio-proxy"
Oct 31 23:36:33.387975 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.387975    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="195909b1-e0bf-4679-84e3-50786cc7eaea" containerName="istio-proxy"
Oct 31 23:36:33.388082 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.388029    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="195909b1-e0bf-4679-84e3-50786cc7eaea" containerName="istio-proxy"
Oct 31 23:36:33.390835 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.390817    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4619f-6dbc6c67dp8772"
Oct 31 23:36:33.402906 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.402882    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c4619f-6dbc6c67dp8772"]
Oct 31 23:36:33.485626 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.485590    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bfe4539-8946-4dfd-9ad7-cc0937c11139-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c4619f-77c85qzrml\" (UID: \"1bfe4539-8946-4dfd-9ad7-cc0937c11139\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4619f-77c85qzrml"
Oct 31 23:36:33.485850 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.485644    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtgkb\" (UniqueName: \"kubernetes.io/projected/1bfe4539-8946-4dfd-9ad7-cc0937c11139-kube-api-access-vtgkb\") pod \"payload-pre-processing-e2e-worker-single-main-c4619f-77c85qzrml\" (UID: \"1bfe4539-8946-4dfd-9ad7-cc0937c11139\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4619f-77c85qzrml"
Oct 31 23:36:33.586475 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.586438    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtgkb\" (UniqueName: \"kubernetes.io/projected/1bfe4539-8946-4dfd-9ad7-cc0937c11139-kube-api-access-vtgkb\") pod \"payload-pre-processing-e2e-worker-single-main-c4619f-77c85qzrml\" (UID: \"1bfe4539-8946-4dfd-9ad7-cc0937c11139\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4619f-77c85qzrml"
Oct 31 23:36:33.586922 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.586511    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn6bn\" (UniqueName: \"kubernetes.io/projected/64867f96-960c-4b1f-9d35-88057c41f66a-kube-api-access-xn6bn\") pod \"payload-processing-e2e-worker-single-main-c4619f-6dbc6c67dp8772\" (UID: \"64867f96-960c-4b1f-9d35-88057c41f66a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4619f-6dbc6c67dp8772"
Oct 31 23:36:33.586922 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.586576    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bfe4539-8946-4dfd-9ad7-cc0937c11139-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c4619f-77c85qzrml\" (UID: \"1bfe4539-8946-4dfd-9ad7-cc0937c11139\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4619f-77c85qzrml"
Oct 31 23:36:33.586922 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.586595    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64867f96-960c-4b1f-9d35-88057c41f66a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c4619f-6dbc6c67dp8772\" (UID: \"64867f96-960c-4b1f-9d35-88057c41f66a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4619f-6dbc6c67dp8772"
Oct 31 23:36:33.587402 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.587351    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bfe4539-8946-4dfd-9ad7-cc0937c11139-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c4619f-77c85qzrml\" (UID: \"1bfe4539-8946-4dfd-9ad7-cc0937c11139\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4619f-77c85qzrml"
Oct 31 23:36:33.597642 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.597618    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtgkb\" (UniqueName: \"kubernetes.io/projected/1bfe4539-8946-4dfd-9ad7-cc0937c11139-kube-api-access-vtgkb\") pod \"payload-pre-processing-e2e-worker-single-main-c4619f-77c85qzrml\" (UID: \"1bfe4539-8946-4dfd-9ad7-cc0937c11139\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4619f-77c85qzrml"
Oct 31 23:36:33.667823 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.667788    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4619f-77c85qzrml"
Oct 31 23:36:33.687344 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.687299    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64867f96-960c-4b1f-9d35-88057c41f66a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c4619f-6dbc6c67dp8772\" (UID: \"64867f96-960c-4b1f-9d35-88057c41f66a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4619f-6dbc6c67dp8772"
Oct 31 23:36:33.687494 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.687442    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn6bn\" (UniqueName: \"kubernetes.io/projected/64867f96-960c-4b1f-9d35-88057c41f66a-kube-api-access-xn6bn\") pod \"payload-processing-e2e-worker-single-main-c4619f-6dbc6c67dp8772\" (UID: \"64867f96-960c-4b1f-9d35-88057c41f66a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4619f-6dbc6c67dp8772"
Oct 31 23:36:33.688351 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.688323    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64867f96-960c-4b1f-9d35-88057c41f66a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c4619f-6dbc6c67dp8772\" (UID: \"64867f96-960c-4b1f-9d35-88057c41f66a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4619f-6dbc6c67dp8772"
Oct 31 23:36:33.699417 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.699388    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn6bn\" (UniqueName: \"kubernetes.io/projected/64867f96-960c-4b1f-9d35-88057c41f66a-kube-api-access-xn6bn\") pod \"payload-processing-e2e-worker-single-main-c4619f-6dbc6c67dp8772\" (UID: \"64867f96-960c-4b1f-9d35-88057c41f66a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4619f-6dbc6c67dp8772"
Oct 31 23:36:33.704184 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.704164    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4619f-6dbc6c67dp8772"
Oct 31 23:36:33.946613 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:33.946533    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4619f-77c85qzrml"]
Oct 31 23:36:33.949788 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:36:33.949754    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bfe4539_8946_4dfd_9ad7_cc0937c11139.slice/crio-884a6e153348861ba380f1dd6465d941fc7a7bd95fa2630fbb7299d149b628b1 WatchSource:0}: Error finding container 884a6e153348861ba380f1dd6465d941fc7a7bd95fa2630fbb7299d149b628b1: Status 404 returned error can't find the container with id 884a6e153348861ba380f1dd6465d941fc7a7bd95fa2630fbb7299d149b628b1
Oct 31 23:36:34.003117 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:34.003095    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c4619f-6dbc6c67dp8772"]
Oct 31 23:36:34.006193 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:36:34.006168    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64867f96_960c_4b1f_9d35_88057c41f66a.slice/crio-b5983533c9125e31ed8a9ea57375ae5c9c340b0fde8b1a437deaeb1b9e6cad22 WatchSource:0}: Error finding container b5983533c9125e31ed8a9ea57375ae5c9c340b0fde8b1a437deaeb1b9e6cad22: Status 404 returned error can't find the container with id b5983533c9125e31ed8a9ea57375ae5c9c340b0fde8b1a437deaeb1b9e6cad22
Oct 31 23:36:34.366873 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:34.366787    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4619f-6dbc6c67dp8772" event={"ID":"64867f96-960c-4b1f-9d35-88057c41f66a","Type":"ContainerStarted","Data":"2eedec1113f35e3169f1a2daa9a316a7470b430669219c68f01e96da3dc97d25"}
Oct 31 23:36:34.366873 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:34.366841    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4619f-6dbc6c67dp8772" event={"ID":"64867f96-960c-4b1f-9d35-88057c41f66a","Type":"ContainerStarted","Data":"b5983533c9125e31ed8a9ea57375ae5c9c340b0fde8b1a437deaeb1b9e6cad22"}
Oct 31 23:36:34.367121 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:34.366955    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4619f-6dbc6c67dp8772"
Oct 31 23:36:34.368300 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:34.368276    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4619f-77c85qzrml" event={"ID":"1bfe4539-8946-4dfd-9ad7-cc0937c11139","Type":"ContainerStarted","Data":"3a1da3aa72c2977e4ba5a01fd6395279a8d4ef8e1cd1b617a5f8a2da96b455bb"}
Oct 31 23:36:34.368466 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:34.368326    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4619f-77c85qzrml"
Oct 31 23:36:34.368466 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:34.368348    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4619f-77c85qzrml" event={"ID":"1bfe4539-8946-4dfd-9ad7-cc0937c11139","Type":"ContainerStarted","Data":"884a6e153348861ba380f1dd6465d941fc7a7bd95fa2630fbb7299d149b628b1"}
Oct 31 23:36:34.383838 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:34.383799    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4619f-6dbc6c67dp8772" podStartSLOduration=1.383788567 podStartE2EDuration="1.383788567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:36:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:36:34.383134038 +0000 UTC m=+2813.130323540" watchObservedRunningTime="2026-10-31 23:36:34.383788567 +0000 UTC m=+2813.130978072"
Oct 31 23:36:34.400261 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:34.400222    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4619f-77c85qzrml" podStartSLOduration=1.4002105249999999 podStartE2EDuration="1.400210525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:36:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:36:34.398549709 +0000 UTC m=+2813.145739212" watchObservedRunningTime="2026-10-31 23:36:34.400210525 +0000 UTC m=+2813.147400030"
Oct 31 23:36:34.795782 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:34.795744    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"]
Oct 31 23:36:34.796246 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:34.795991    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr" podUID="14a4227c-cbbd-4193-81e9-56287613d98b" containerName="istio-proxy" containerID="cri-o://dbb43a7620a5ec4d93fc790a8ebaa7149fb982445c0e42082db6b9eb7e89542b" gracePeriod=30
Oct 31 23:36:34.797865 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:34.797722    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"]
Oct 31 23:36:34.798116 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:34.798074    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg" podUID="201d729f-baac-4eb9-bcbc-497ad72370b9" containerName="istio-proxy" containerID="cri-o://b5ca80e09dddaf0f88f881e951e08a88b79217f5d0df1e2ca5a4279cf7231c71" gracePeriod=30
Oct 31 23:36:34.799167 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:34.799134    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"]
Oct 31 23:36:34.799405 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:34.799319    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f" podUID="86a238da-6602-4f4b-a300-be4347d907e5" containerName="istio-proxy" containerID="cri-o://f80ca256dc927e3772f9e02f43d640370011428d71d9a44d4d3e155ded135e57" gracePeriod=30
Oct 31 23:36:36.086767 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:36.086732    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:36:40.049617 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.049589    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:36:40.088019 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.087988    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:36:40.091753 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.091732    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:36:40.134984 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.134952    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-credential-socket\") pod \"201d729f-baac-4eb9-bcbc-497ad72370b9\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") "
Oct 31 23:36:40.135138 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.134995    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-istio-data\") pod \"14a4227c-cbbd-4193-81e9-56287613d98b\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") "
Oct 31 23:36:40.135138 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.135026    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/201d729f-baac-4eb9-bcbc-497ad72370b9-istio-token\" (UniqueName: \"kubernetes.io/projected/201d729f-baac-4eb9-bcbc-497ad72370b9-istio-token\") pod \"201d729f-baac-4eb9-bcbc-497ad72370b9\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") "
Oct 31 23:36:40.135138 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.135058    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/201d729f-baac-4eb9-bcbc-497ad72370b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/201d729f-baac-4eb9-bcbc-497ad72370b9-istiod-ca-cert\") pod \"201d729f-baac-4eb9-bcbc-497ad72370b9\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") "
Oct 31 23:36:40.135138 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.135084    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-istio-envoy\") pod \"201d729f-baac-4eb9-bcbc-497ad72370b9\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") "
Oct 31 23:36:40.135138 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.135110    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-credential-socket\") pod \"14a4227c-cbbd-4193-81e9-56287613d98b\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") "
Oct 31 23:36:40.135953 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.135139    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/201d729f-baac-4eb9-bcbc-497ad72370b9-kube-api-access-5pttg\" (UniqueName: \"kubernetes.io/projected/201d729f-baac-4eb9-bcbc-497ad72370b9-kube-api-access-5pttg\") pod \"201d729f-baac-4eb9-bcbc-497ad72370b9\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") "
Oct 31 23:36:40.135953 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.135167    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86a238da-6602-4f4b-a300-be4347d907e5-istio-token\" (UniqueName: \"kubernetes.io/projected/86a238da-6602-4f4b-a300-be4347d907e5-istio-token\") pod \"86a238da-6602-4f4b-a300-be4347d907e5\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") "
Oct 31 23:36:40.135953 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.135198    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-workload-certs\") pod \"86a238da-6602-4f4b-a300-be4347d907e5\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") "
Oct 31 23:36:40.135953 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.135215    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-credential-socket" pod "201d729f-baac-4eb9-bcbc-497ad72370b9" (UID: "201d729f-baac-4eb9-bcbc-497ad72370b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:36:40.135953 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.135223    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-istio-data\") pod \"201d729f-baac-4eb9-bcbc-497ad72370b9\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") "
Oct 31 23:36:40.135953 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.135236    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-istio-data" pod "14a4227c-cbbd-4193-81e9-56287613d98b" (UID: "14a4227c-cbbd-4193-81e9-56287613d98b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:36:40.135953 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.135259    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-workload-socket\") pod \"86a238da-6602-4f4b-a300-be4347d907e5\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") "
Oct 31 23:36:40.135953 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.135289    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-istio-envoy\") pod \"86a238da-6602-4f4b-a300-be4347d907e5\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") "
Oct 31 23:36:40.135953 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.135381    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14a4227c-cbbd-4193-81e9-56287613d98b-kube-api-access-98hl9\" (UniqueName: \"kubernetes.io/projected/14a4227c-cbbd-4193-81e9-56287613d98b-kube-api-access-98hl9\") pod \"14a4227c-cbbd-4193-81e9-56287613d98b\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") "
Oct 31 23:36:40.135953 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.135427    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-credential-socket\") pod \"86a238da-6602-4f4b-a300-be4347d907e5\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") "
Oct 31 23:36:40.135953 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.135446    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-credential-socket" pod "14a4227c-cbbd-4193-81e9-56287613d98b" (UID: "14a4227c-cbbd-4193-81e9-56287613d98b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:36:40.135953 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.135460    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-workload-socket\") pod \"201d729f-baac-4eb9-bcbc-497ad72370b9\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") "
Oct 31 23:36:40.135953 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.135494    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86a238da-6602-4f4b-a300-be4347d907e5-kube-api-access-khh2n\" (UniqueName: \"kubernetes.io/projected/86a238da-6602-4f4b-a300-be4347d907e5-kube-api-access-khh2n\") pod \"86a238da-6602-4f4b-a300-be4347d907e5\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") "
Oct 31 23:36:40.135953 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.135521    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14a4227c-cbbd-4193-81e9-56287613d98b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14a4227c-cbbd-4193-81e9-56287613d98b-istio-podinfo\") pod \"14a4227c-cbbd-4193-81e9-56287613d98b\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") "
Oct 31 23:36:40.135953 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.135560    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-istio-data" pod "201d729f-baac-4eb9-bcbc-497ad72370b9" (UID: "201d729f-baac-4eb9-bcbc-497ad72370b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:36:40.135953 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.135570    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/201d729f-baac-4eb9-bcbc-497ad72370b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/201d729f-baac-4eb9-bcbc-497ad72370b9-istio-podinfo\") pod \"201d729f-baac-4eb9-bcbc-497ad72370b9\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") "
Oct 31 23:36:40.135953 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.135601    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-istio-data\") pod \"86a238da-6602-4f4b-a300-be4347d907e5\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") "
Oct 31 23:36:40.135953 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.135629    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-workload-certs\") pod \"14a4227c-cbbd-4193-81e9-56287613d98b\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") "
Oct 31 23:36:40.135953 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.135665    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-workload-socket\") pod \"14a4227c-cbbd-4193-81e9-56287613d98b\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") "
Oct 31 23:36:40.135953 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.135681    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/201d729f-baac-4eb9-bcbc-497ad72370b9-istiod-ca-cert" pod "201d729f-baac-4eb9-bcbc-497ad72370b9" (UID: "201d729f-baac-4eb9-bcbc-497ad72370b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:36:40.135953 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.135694    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-istio-envoy\") pod \"14a4227c-cbbd-4193-81e9-56287613d98b\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") "
Oct 31 23:36:40.135953 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.135752    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-workload-certs\") pod \"201d729f-baac-4eb9-bcbc-497ad72370b9\" (UID: \"201d729f-baac-4eb9-bcbc-497ad72370b9\") "
Oct 31 23:36:40.138563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.136188    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-workload-certs" pod "86a238da-6602-4f4b-a300-be4347d907e5" (UID: "86a238da-6602-4f4b-a300-be4347d907e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:36:40.138563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.136662    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:36:40.138563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.136713    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:36:40.138563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.136732    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/201d729f-baac-4eb9-bcbc-497ad72370b9-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:36:40.138563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.136748    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:36:40.138563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.136762    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:36:40.138563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.136964    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-workload-certs" pod "201d729f-baac-4eb9-bcbc-497ad72370b9" (UID: "201d729f-baac-4eb9-bcbc-497ad72370b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:36:40.138563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.137151    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-credential-socket" pod "86a238da-6602-4f4b-a300-be4347d907e5" (UID: "86a238da-6602-4f4b-a300-be4347d907e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:36:40.138563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.137986    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-workload-socket" pod "201d729f-baac-4eb9-bcbc-497ad72370b9" (UID: "201d729f-baac-4eb9-bcbc-497ad72370b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:36:40.138563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.138015    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-workload-socket" pod "14a4227c-cbbd-4193-81e9-56287613d98b" (UID: "14a4227c-cbbd-4193-81e9-56287613d98b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:36:40.138563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.138063    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-workload-socket" pod "86a238da-6602-4f4b-a300-be4347d907e5" (UID: "86a238da-6602-4f4b-a300-be4347d907e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:36:40.138563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.138118    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-istio-data" pod "86a238da-6602-4f4b-a300-be4347d907e5" (UID: "86a238da-6602-4f4b-a300-be4347d907e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:36:40.138563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.138327    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-workload-certs" pod "14a4227c-cbbd-4193-81e9-56287613d98b" (UID: "14a4227c-cbbd-4193-81e9-56287613d98b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:36:40.139783 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.139759    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-istio-envoy" pod "14a4227c-cbbd-4193-81e9-56287613d98b" (UID: "14a4227c-cbbd-4193-81e9-56287613d98b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:36:40.140235 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.140185    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a4227c-cbbd-4193-81e9-56287613d98b-kube-api-access-98hl9" pod "14a4227c-cbbd-4193-81e9-56287613d98b" (UID: "14a4227c-cbbd-4193-81e9-56287613d98b"). InnerVolumeSpecName "kube-api-access-98hl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:36:40.140417 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.140392    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-istio-envoy" pod "201d729f-baac-4eb9-bcbc-497ad72370b9" (UID: "201d729f-baac-4eb9-bcbc-497ad72370b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:36:40.140701 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.140677    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86a238da-6602-4f4b-a300-be4347d907e5-istio-token" pod "86a238da-6602-4f4b-a300-be4347d907e5" (UID: "86a238da-6602-4f4b-a300-be4347d907e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:36:40.140817 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.140781    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/201d729f-baac-4eb9-bcbc-497ad72370b9-istio-podinfo" pod "201d729f-baac-4eb9-bcbc-497ad72370b9" (UID: "201d729f-baac-4eb9-bcbc-497ad72370b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:36:40.140974 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.140885    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-istio-envoy" pod "86a238da-6602-4f4b-a300-be4347d907e5" (UID: "86a238da-6602-4f4b-a300-be4347d907e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:36:40.140974 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.140928    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/201d729f-baac-4eb9-bcbc-497ad72370b9-kube-api-access-5pttg" pod "201d729f-baac-4eb9-bcbc-497ad72370b9" (UID: "201d729f-baac-4eb9-bcbc-497ad72370b9"). InnerVolumeSpecName "kube-api-access-5pttg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:36:40.141090 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.141048    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/201d729f-baac-4eb9-bcbc-497ad72370b9-istio-token" pod "201d729f-baac-4eb9-bcbc-497ad72370b9" (UID: "201d729f-baac-4eb9-bcbc-497ad72370b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:36:40.141657 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.141634    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14a4227c-cbbd-4193-81e9-56287613d98b-istio-podinfo" pod "14a4227c-cbbd-4193-81e9-56287613d98b" (UID: "14a4227c-cbbd-4193-81e9-56287613d98b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:36:40.142507 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.142487    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86a238da-6602-4f4b-a300-be4347d907e5-kube-api-access-khh2n" pod "86a238da-6602-4f4b-a300-be4347d907e5" (UID: "86a238da-6602-4f4b-a300-be4347d907e5"). InnerVolumeSpecName "kube-api-access-khh2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:36:40.237169 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.237137    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86a238da-6602-4f4b-a300-be4347d907e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86a238da-6602-4f4b-a300-be4347d907e5-istio-podinfo\") pod \"86a238da-6602-4f4b-a300-be4347d907e5\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") "
Oct 31 23:36:40.237380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.237185    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14a4227c-cbbd-4193-81e9-56287613d98b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14a4227c-cbbd-4193-81e9-56287613d98b-istiod-ca-cert\") pod \"14a4227c-cbbd-4193-81e9-56287613d98b\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") "
Oct 31 23:36:40.237380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.237209    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14a4227c-cbbd-4193-81e9-56287613d98b-istio-token\" (UniqueName: \"kubernetes.io/projected/14a4227c-cbbd-4193-81e9-56287613d98b-istio-token\") pod \"14a4227c-cbbd-4193-81e9-56287613d98b\" (UID: \"14a4227c-cbbd-4193-81e9-56287613d98b\") "
Oct 31 23:36:40.237380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.237240    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86a238da-6602-4f4b-a300-be4347d907e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86a238da-6602-4f4b-a300-be4347d907e5-istiod-ca-cert\") pod \"86a238da-6602-4f4b-a300-be4347d907e5\" (UID: \"86a238da-6602-4f4b-a300-be4347d907e5\") "
Oct 31 23:36:40.237563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.237529    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98hl9\" (UniqueName: \"kubernetes.io/projected/14a4227c-cbbd-4193-81e9-56287613d98b-kube-api-access-98hl9\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:36:40.237563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.237550    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:36:40.237683 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.237564    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:36:40.237683 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.237580    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khh2n\" (UniqueName: \"kubernetes.io/projected/86a238da-6602-4f4b-a300-be4347d907e5-kube-api-access-khh2n\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:36:40.237683 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.237595    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14a4227c-cbbd-4193-81e9-56287613d98b-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:36:40.237683 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.237611    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/201d729f-baac-4eb9-bcbc-497ad72370b9-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:36:40.237683 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.237625    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:36:40.237683 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.237641    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:36:40.237683 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.237647    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86a238da-6602-4f4b-a300-be4347d907e5-istiod-ca-cert" pod "86a238da-6602-4f4b-a300-be4347d907e5" (UID: "86a238da-6602-4f4b-a300-be4347d907e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:36:40.237927 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.237701    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:36:40.237927 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.237703    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a4227c-cbbd-4193-81e9-56287613d98b-istiod-ca-cert" pod "14a4227c-cbbd-4193-81e9-56287613d98b" (UID: "14a4227c-cbbd-4193-81e9-56287613d98b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:36:40.237927 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.237731    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14a4227c-cbbd-4193-81e9-56287613d98b-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:36:40.237927 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.237746    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:36:40.237927 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.237760    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/201d729f-baac-4eb9-bcbc-497ad72370b9-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:36:40.237927 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.237774    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/201d729f-baac-4eb9-bcbc-497ad72370b9-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:36:40.237927 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.237788    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pttg\" (UniqueName: \"kubernetes.io/projected/201d729f-baac-4eb9-bcbc-497ad72370b9-kube-api-access-5pttg\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:36:40.237927 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.237802    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86a238da-6602-4f4b-a300-be4347d907e5-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:36:40.237927 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.237816    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:36:40.237927 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.237830    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:36:40.237927 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.237843    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86a238da-6602-4f4b-a300-be4347d907e5-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:36:40.239294 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.239273    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86a238da-6602-4f4b-a300-be4347d907e5-istio-podinfo" pod "86a238da-6602-4f4b-a300-be4347d907e5" (UID: "86a238da-6602-4f4b-a300-be4347d907e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:36:40.239394 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.239339    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a4227c-cbbd-4193-81e9-56287613d98b-istio-token" pod "14a4227c-cbbd-4193-81e9-56287613d98b" (UID: "14a4227c-cbbd-4193-81e9-56287613d98b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:36:40.338458 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.338383    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86a238da-6602-4f4b-a300-be4347d907e5-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:36:40.338458 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.338408    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14a4227c-cbbd-4193-81e9-56287613d98b-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:36:40.338458 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.338417    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14a4227c-cbbd-4193-81e9-56287613d98b-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:36:40.338458 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.338425    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86a238da-6602-4f4b-a300-be4347d907e5-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:36:40.396594 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.396564    4301 generic.go:356] "Generic (PLEG): container finished" podID="14a4227c-cbbd-4193-81e9-56287613d98b" containerID="dbb43a7620a5ec4d93fc790a8ebaa7149fb982445c0e42082db6b9eb7e89542b" exitCode=0
Oct 31 23:36:40.396778 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.396614    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"
Oct 31 23:36:40.396778 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.396641    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr" event={"ID":"14a4227c-cbbd-4193-81e9-56287613d98b","Type":"ContainerDied","Data":"dbb43a7620a5ec4d93fc790a8ebaa7149fb982445c0e42082db6b9eb7e89542b"}
Oct 31 23:36:40.396778 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.396689    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr" event={"ID":"14a4227c-cbbd-4193-81e9-56287613d98b","Type":"ContainerDied","Data":"fd884226d6e3e24008d437fbbaf8b23338a3bce3152c7d520bc883e48ec4c4a2"}
Oct 31 23:36:40.396778 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.396719    4301 scope.go:122] "RemoveContainer" containerID="dbb43a7620a5ec4d93fc790a8ebaa7149fb982445c0e42082db6b9eb7e89542b"
Oct 31 23:36:40.398124 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.398105    4301 generic.go:356] "Generic (PLEG): container finished" podID="201d729f-baac-4eb9-bcbc-497ad72370b9" containerID="b5ca80e09dddaf0f88f881e951e08a88b79217f5d0df1e2ca5a4279cf7231c71" exitCode=0
Oct 31 23:36:40.398218 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.398130    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg" event={"ID":"201d729f-baac-4eb9-bcbc-497ad72370b9","Type":"ContainerDied","Data":"b5ca80e09dddaf0f88f881e951e08a88b79217f5d0df1e2ca5a4279cf7231c71"}
Oct 31 23:36:40.398218 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.398160    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"
Oct 31 23:36:40.398218 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.398175    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg" event={"ID":"201d729f-baac-4eb9-bcbc-497ad72370b9","Type":"ContainerDied","Data":"1780aa8a4725810f417f580f773c1f88d295f267242e83128492cdf15d3e3ec7"}
Oct 31 23:36:40.409677 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.399632    4301 generic.go:356] "Generic (PLEG): container finished" podID="86a238da-6602-4f4b-a300-be4347d907e5" containerID="f80ca256dc927e3772f9e02f43d640370011428d71d9a44d4d3e155ded135e57" exitCode=0
Oct 31 23:36:40.409677 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.399665    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f" event={"ID":"86a238da-6602-4f4b-a300-be4347d907e5","Type":"ContainerDied","Data":"f80ca256dc927e3772f9e02f43d640370011428d71d9a44d4d3e155ded135e57"}
Oct 31 23:36:40.409677 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.399714    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"
Oct 31 23:36:40.409677 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.399721    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f" event={"ID":"86a238da-6602-4f4b-a300-be4347d907e5","Type":"ContainerDied","Data":"5922f22e070b4de6f675a085e3fe4631a0b855679b05ab229ea0026f38b7bb9b"}
Oct 31 23:36:40.414582 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.414564    4301 scope.go:122] "RemoveContainer" containerID="dbb43a7620a5ec4d93fc790a8ebaa7149fb982445c0e42082db6b9eb7e89542b"
Oct 31 23:36:40.414849 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:36:40.414831    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbb43a7620a5ec4d93fc790a8ebaa7149fb982445c0e42082db6b9eb7e89542b\": container with ID starting with dbb43a7620a5ec4d93fc790a8ebaa7149fb982445c0e42082db6b9eb7e89542b not found: ID does not exist" containerID="dbb43a7620a5ec4d93fc790a8ebaa7149fb982445c0e42082db6b9eb7e89542b"
Oct 31 23:36:40.414895 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.414855    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbb43a7620a5ec4d93fc790a8ebaa7149fb982445c0e42082db6b9eb7e89542b"} err="failed to get container status \"dbb43a7620a5ec4d93fc790a8ebaa7149fb982445c0e42082db6b9eb7e89542b\": rpc error: code = NotFound desc = could not find container \"dbb43a7620a5ec4d93fc790a8ebaa7149fb982445c0e42082db6b9eb7e89542b\": container with ID starting with dbb43a7620a5ec4d93fc790a8ebaa7149fb982445c0e42082db6b9eb7e89542b not found: ID does not exist"
Oct 31 23:36:40.414895 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.414871    4301 scope.go:122] "RemoveContainer" containerID="b5ca80e09dddaf0f88f881e951e08a88b79217f5d0df1e2ca5a4279cf7231c71"
Oct 31 23:36:40.422648 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.422627    4301 scope.go:122] "RemoveContainer" containerID="b5ca80e09dddaf0f88f881e951e08a88b79217f5d0df1e2ca5a4279cf7231c71"
Oct 31 23:36:40.422930 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:36:40.422913    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5ca80e09dddaf0f88f881e951e08a88b79217f5d0df1e2ca5a4279cf7231c71\": container with ID starting with b5ca80e09dddaf0f88f881e951e08a88b79217f5d0df1e2ca5a4279cf7231c71 not found: ID does not exist" containerID="b5ca80e09dddaf0f88f881e951e08a88b79217f5d0df1e2ca5a4279cf7231c71"
Oct 31 23:36:40.422982 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.422939    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5ca80e09dddaf0f88f881e951e08a88b79217f5d0df1e2ca5a4279cf7231c71"} err="failed to get container status \"b5ca80e09dddaf0f88f881e951e08a88b79217f5d0df1e2ca5a4279cf7231c71\": rpc error: code = NotFound desc = could not find container \"b5ca80e09dddaf0f88f881e951e08a88b79217f5d0df1e2ca5a4279cf7231c71\": container with ID starting with b5ca80e09dddaf0f88f881e951e08a88b79217f5d0df1e2ca5a4279cf7231c71 not found: ID does not exist"
Oct 31 23:36:40.422982 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.422954    4301 scope.go:122] "RemoveContainer" containerID="f80ca256dc927e3772f9e02f43d640370011428d71d9a44d4d3e155ded135e57"
Oct 31 23:36:40.425068 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.425046    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"]
Oct 31 23:36:40.431293 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.431274    4301 scope.go:122] "RemoveContainer" containerID="f80ca256dc927e3772f9e02f43d640370011428d71d9a44d4d3e155ded135e57"
Oct 31 23:36:40.431624 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:36:40.431599    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f80ca256dc927e3772f9e02f43d640370011428d71d9a44d4d3e155ded135e57\": container with ID starting with f80ca256dc927e3772f9e02f43d640370011428d71d9a44d4d3e155ded135e57 not found: ID does not exist" containerID="f80ca256dc927e3772f9e02f43d640370011428d71d9a44d4d3e155ded135e57"
Oct 31 23:36:40.431693 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.431633    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f80ca256dc927e3772f9e02f43d640370011428d71d9a44d4d3e155ded135e57"} err="failed to get container status \"f80ca256dc927e3772f9e02f43d640370011428d71d9a44d4d3e155ded135e57\": rpc error: code = NotFound desc = could not find container \"f80ca256dc927e3772f9e02f43d640370011428d71d9a44d4d3e155ded135e57\": container with ID starting with f80ca256dc927e3772f9e02f43d640370011428d71d9a44d4d3e155ded135e57 not found: ID does not exist"
Oct 31 23:36:40.431807 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.431793    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-922mr"]
Oct 31 23:36:40.442218 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.442194    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"]
Oct 31 23:36:40.447781 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.447758    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-v559f"]
Oct 31 23:36:40.458440 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.458421    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"]
Oct 31 23:36:40.462697 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:40.462673    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lprvg"]
Oct 31 23:36:41.402172 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:41.402131    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14a4227c-cbbd-4193-81e9-56287613d98b" path="/var/lib/kubelet/pods/14a4227c-cbbd-4193-81e9-56287613d98b/volumes"
Oct 31 23:36:41.402801 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:41.402783    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="201d729f-baac-4eb9-bcbc-497ad72370b9" path="/var/lib/kubelet/pods/201d729f-baac-4eb9-bcbc-497ad72370b9/volumes"
Oct 31 23:36:41.403264 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:41.403244    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86a238da-6602-4f4b-a300-be4347d907e5" path="/var/lib/kubelet/pods/86a238da-6602-4f4b-a300-be4347d907e5/volumes"
Oct 31 23:36:45.375192 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:45.375162    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4619f-77c85qzrml"
Oct 31 23:36:45.375619 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:45.375227    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4619f-6dbc6c67dp8772"
Oct 31 23:36:49.024966 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.024930    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv"]
Oct 31 23:36:49.025467 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.025246    4301 cpu_manager.go:414] "containerMap: removing container" podUID="86a238da-6602-4f4b-a300-be4347d907e5" containerName="istio-proxy"
Oct 31 23:36:49.025467 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.025260    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86a238da-6602-4f4b-a300-be4347d907e5" containerName="istio-proxy"
Oct 31 23:36:49.025467 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.025280    4301 cpu_manager.go:414] "containerMap: removing container" podUID="201d729f-baac-4eb9-bcbc-497ad72370b9" containerName="istio-proxy"
Oct 31 23:36:49.025467 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.025285    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="201d729f-baac-4eb9-bcbc-497ad72370b9" containerName="istio-proxy"
Oct 31 23:36:49.025467 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.025299    4301 cpu_manager.go:414] "containerMap: removing container" podUID="14a4227c-cbbd-4193-81e9-56287613d98b" containerName="istio-proxy"
Oct 31 23:36:49.025467 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.025306    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14a4227c-cbbd-4193-81e9-56287613d98b" containerName="istio-proxy"
Oct 31 23:36:49.025467 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.025414    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="86a238da-6602-4f4b-a300-be4347d907e5" containerName="istio-proxy"
Oct 31 23:36:49.025467 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.025426    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="201d729f-baac-4eb9-bcbc-497ad72370b9" containerName="istio-proxy"
Oct 31 23:36:49.025467 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.025433    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="14a4227c-cbbd-4193-81e9-56287613d98b" containerName="istio-proxy"
Oct 31 23:36:49.029705 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.029668    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv"
Oct 31 23:36:49.035522 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.035498    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-c4619f\"/\"kube-root-ca.crt\""
Oct 31 23:36:49.035522 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.035510    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-c4619f\"/\"default-dockercfg-qqcln\""
Oct 31 23:36:49.035754 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.035533    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-c4619f\"/\"openshift-service-ca.crt\""
Oct 31 23:36:49.035754 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.035602    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-c4619f\"/\"simulator-single-main-c4619f-kserve-self-signed-certs\""
Oct 31 23:36:49.040005 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.039971    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv"]
Oct 31 23:36:49.109474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.109434    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-tls-certs\") pod \"simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv\" (UID: \"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d\") " pod="e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv"
Oct 31 23:36:49.109681 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.109499    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvlbn\" (UniqueName: \"kubernetes.io/projected/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-kube-api-access-qvlbn\") pod \"simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv\" (UID: \"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d\") " pod="e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv"
Oct 31 23:36:49.109681 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.109649    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-model-cache\") pod \"simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv\" (UID: \"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d\") " pod="e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv"
Oct 31 23:36:49.109796 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.109711    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-dshm\") pod \"simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv\" (UID: \"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d\") " pod="e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv"
Oct 31 23:36:49.109856 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.109797    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-tmp-dir\") pod \"simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv\" (UID: \"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d\") " pod="e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv"
Oct 31 23:36:49.109856 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.109844    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-home\") pod \"simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv\" (UID: \"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d\") " pod="e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv"
Oct 31 23:36:49.211385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.211329    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-tls-certs\") pod \"simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv\" (UID: \"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d\") " pod="e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv"
Oct 31 23:36:49.211543 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.211397    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvlbn\" (UniqueName: \"kubernetes.io/projected/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-kube-api-access-qvlbn\") pod \"simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv\" (UID: \"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d\") " pod="e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv"
Oct 31 23:36:49.211543 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.211452    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-model-cache\") pod \"simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv\" (UID: \"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d\") " pod="e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv"
Oct 31 23:36:49.211651 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.211562    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-dshm\") pod \"simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv\" (UID: \"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d\") " pod="e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv"
Oct 31 23:36:49.211651 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.211589    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-tmp-dir\") pod \"simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv\" (UID: \"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d\") " pod="e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv"
Oct 31 23:36:49.211651 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.211623    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-home\") pod \"simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv\" (UID: \"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d\") " pod="e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv"
Oct 31 23:36:49.212053 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.212027    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-model-cache\") pod \"simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv\" (UID: \"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d\") " pod="e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv"
Oct 31 23:36:49.212167 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.212053    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-home\") pod \"simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv\" (UID: \"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d\") " pod="e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv"
Oct 31 23:36:49.212221 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.212207    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-tmp-dir\") pod \"simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv\" (UID: \"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d\") " pod="e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv"
Oct 31 23:36:49.214090 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.214070    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-tls-certs\") pod \"simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv\" (UID: \"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d\") " pod="e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv"
Oct 31 23:36:49.214183 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.214107    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-dshm\") pod \"simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv\" (UID: \"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d\") " pod="e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv"
Oct 31 23:36:49.222954 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.222935    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvlbn\" (UniqueName: \"kubernetes.io/projected/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-kube-api-access-qvlbn\") pod \"simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv\" (UID: \"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d\") " pod="e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv"
Oct 31 23:36:49.342195 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.342097    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv"
Oct 31 23:36:49.617653 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.617571    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv"]
Oct 31 23:36:49.621259 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:36:49.621238    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f1ac3f1_e2c9_4e5e_a25a_496af40f046d.slice/crio-3571b78c167ae86732757c95075b2b0d174f8b432e123bd885abe86d79612601 WatchSource:0}: Error finding container 3571b78c167ae86732757c95075b2b0d174f8b432e123bd885abe86d79612601: Status 404 returned error can't find the container with id 3571b78c167ae86732757c95075b2b0d174f8b432e123bd885abe86d79612601
Oct 31 23:36:49.798848 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.798813    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"]
Oct 31 23:36:49.802003 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.801983    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"
Oct 31 23:36:49.813933 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.813913    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 23:36:49.816320 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.816301    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-nf6c6\""
Oct 31 23:36:49.818849 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.818829    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"]
Oct 31 23:36:49.918036 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.918002    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99873d5d-87c2-423a-9a44-1b3fc83d0b46-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-s6nvl\" (UID: \"99873d5d-87c2-423a-9a44-1b3fc83d0b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"
Oct 31 23:36:49.918216 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.918067    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltn4r\" (UniqueName: \"kubernetes.io/projected/99873d5d-87c2-423a-9a44-1b3fc83d0b46-kube-api-access-ltn4r\") pod \"maas-default-gateway-openshift-default-6886b96f9-s6nvl\" (UID: \"99873d5d-87c2-423a-9a44-1b3fc83d0b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"
Oct 31 23:36:49.918216 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.918137    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99873d5d-87c2-423a-9a44-1b3fc83d0b46-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-s6nvl\" (UID: \"99873d5d-87c2-423a-9a44-1b3fc83d0b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"
Oct 31 23:36:49.918216 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.918176    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99873d5d-87c2-423a-9a44-1b3fc83d0b46-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-s6nvl\" (UID: \"99873d5d-87c2-423a-9a44-1b3fc83d0b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"
Oct 31 23:36:49.918216 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.918199    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99873d5d-87c2-423a-9a44-1b3fc83d0b46-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-s6nvl\" (UID: \"99873d5d-87c2-423a-9a44-1b3fc83d0b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"
Oct 31 23:36:49.918353 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.918290    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99873d5d-87c2-423a-9a44-1b3fc83d0b46-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-s6nvl\" (UID: \"99873d5d-87c2-423a-9a44-1b3fc83d0b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"
Oct 31 23:36:49.918353 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.918317    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99873d5d-87c2-423a-9a44-1b3fc83d0b46-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-s6nvl\" (UID: \"99873d5d-87c2-423a-9a44-1b3fc83d0b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"
Oct 31 23:36:49.918448 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.918380    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99873d5d-87c2-423a-9a44-1b3fc83d0b46-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-s6nvl\" (UID: \"99873d5d-87c2-423a-9a44-1b3fc83d0b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"
Oct 31 23:36:49.918448 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:49.918403    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99873d5d-87c2-423a-9a44-1b3fc83d0b46-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-s6nvl\" (UID: \"99873d5d-87c2-423a-9a44-1b3fc83d0b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"
Oct 31 23:36:50.019426 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:50.019385    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99873d5d-87c2-423a-9a44-1b3fc83d0b46-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-s6nvl\" (UID: \"99873d5d-87c2-423a-9a44-1b3fc83d0b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"
Oct 31 23:36:50.019616 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:50.019434    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99873d5d-87c2-423a-9a44-1b3fc83d0b46-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-s6nvl\" (UID: \"99873d5d-87c2-423a-9a44-1b3fc83d0b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"
Oct 31 23:36:50.019616 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:50.019472    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99873d5d-87c2-423a-9a44-1b3fc83d0b46-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-s6nvl\" (UID: \"99873d5d-87c2-423a-9a44-1b3fc83d0b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"
Oct 31 23:36:50.019616 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:50.019505    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99873d5d-87c2-423a-9a44-1b3fc83d0b46-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-s6nvl\" (UID: \"99873d5d-87c2-423a-9a44-1b3fc83d0b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"
Oct 31 23:36:50.019616 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:50.019546    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99873d5d-87c2-423a-9a44-1b3fc83d0b46-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-s6nvl\" (UID: \"99873d5d-87c2-423a-9a44-1b3fc83d0b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"
Oct 31 23:36:50.019616 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:50.019592    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltn4r\" (UniqueName: \"kubernetes.io/projected/99873d5d-87c2-423a-9a44-1b3fc83d0b46-kube-api-access-ltn4r\") pod \"maas-default-gateway-openshift-default-6886b96f9-s6nvl\" (UID: \"99873d5d-87c2-423a-9a44-1b3fc83d0b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"
Oct 31 23:36:50.019899 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:50.019623    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99873d5d-87c2-423a-9a44-1b3fc83d0b46-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-s6nvl\" (UID: \"99873d5d-87c2-423a-9a44-1b3fc83d0b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"
Oct 31 23:36:50.019899 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:50.019661    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99873d5d-87c2-423a-9a44-1b3fc83d0b46-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-s6nvl\" (UID: \"99873d5d-87c2-423a-9a44-1b3fc83d0b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"
Oct 31 23:36:50.019899 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:50.019693    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99873d5d-87c2-423a-9a44-1b3fc83d0b46-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-s6nvl\" (UID: \"99873d5d-87c2-423a-9a44-1b3fc83d0b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"
Oct 31 23:36:50.020064 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:50.019935    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99873d5d-87c2-423a-9a44-1b3fc83d0b46-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-s6nvl\" (UID: \"99873d5d-87c2-423a-9a44-1b3fc83d0b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"
Oct 31 23:36:50.020064 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:50.020028    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99873d5d-87c2-423a-9a44-1b3fc83d0b46-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-s6nvl\" (UID: \"99873d5d-87c2-423a-9a44-1b3fc83d0b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"
Oct 31 23:36:50.020064 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:50.020051    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99873d5d-87c2-423a-9a44-1b3fc83d0b46-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-s6nvl\" (UID: \"99873d5d-87c2-423a-9a44-1b3fc83d0b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"
Oct 31 23:36:50.020207 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:50.020101    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99873d5d-87c2-423a-9a44-1b3fc83d0b46-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-s6nvl\" (UID: \"99873d5d-87c2-423a-9a44-1b3fc83d0b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"
Oct 31 23:36:50.020561 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:50.020525    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99873d5d-87c2-423a-9a44-1b3fc83d0b46-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-s6nvl\" (UID: \"99873d5d-87c2-423a-9a44-1b3fc83d0b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"
Oct 31 23:36:50.021987 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:50.021967    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99873d5d-87c2-423a-9a44-1b3fc83d0b46-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-s6nvl\" (UID: \"99873d5d-87c2-423a-9a44-1b3fc83d0b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"
Oct 31 23:36:50.022189 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:50.022169    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99873d5d-87c2-423a-9a44-1b3fc83d0b46-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-s6nvl\" (UID: \"99873d5d-87c2-423a-9a44-1b3fc83d0b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"
Oct 31 23:36:50.030981 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:50.030956    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99873d5d-87c2-423a-9a44-1b3fc83d0b46-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-s6nvl\" (UID: \"99873d5d-87c2-423a-9a44-1b3fc83d0b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"
Oct 31 23:36:50.031255 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:50.031212    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltn4r\" (UniqueName: \"kubernetes.io/projected/99873d5d-87c2-423a-9a44-1b3fc83d0b46-kube-api-access-ltn4r\") pod \"maas-default-gateway-openshift-default-6886b96f9-s6nvl\" (UID: \"99873d5d-87c2-423a-9a44-1b3fc83d0b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"
Oct 31 23:36:50.114145 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:50.114113    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"
Oct 31 23:36:50.399039 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:50.398631    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"]
Oct 31 23:36:50.401485 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:36:50.401460    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99873d5d_87c2_423a_9a44_1b3fc83d0b46.slice/crio-2873071de3cf3fbd036e180a7891bd70282b94b1df19da7649c64632a52631a6 WatchSource:0}: Error finding container 2873071de3cf3fbd036e180a7891bd70282b94b1df19da7649c64632a52631a6: Status 404 returned error can't find the container with id 2873071de3cf3fbd036e180a7891bd70282b94b1df19da7649c64632a52631a6
Oct 31 23:36:50.404107 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:50.404074    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:36:50.404210 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:50.404134    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:36:50.404210 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:50.404163    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:36:50.445438 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:50.445406    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl" event={"ID":"99873d5d-87c2-423a-9a44-1b3fc83d0b46","Type":"ContainerStarted","Data":"2873071de3cf3fbd036e180a7891bd70282b94b1df19da7649c64632a52631a6"}
Oct 31 23:36:50.446929 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:50.446902    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv" event={"ID":"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d","Type":"ContainerStarted","Data":"47be8ace0f7ba239962d48e7d97bfff213bccd20da8d4d03fe33ea99c20b2c72"}
Oct 31 23:36:50.447047 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:50.446939    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv" event={"ID":"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d","Type":"ContainerStarted","Data":"3571b78c167ae86732757c95075b2b0d174f8b432e123bd885abe86d79612601"}
Oct 31 23:36:50.465895 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:50.465839    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv" podStartSLOduration=1.4658217900000001 podStartE2EDuration="1.46582179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:36:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:36:50.464117271 +0000 UTC m=+2829.211306773" watchObservedRunningTime="2026-10-31 23:36:50.46582179 +0000 UTC m=+2829.213011301"
Oct 31 23:36:50.913305 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:50.913269    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:36:51.343570 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:51.343466    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv"
Oct 31 23:36:51.459504 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:51.459457    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl" event={"ID":"99873d5d-87c2-423a-9a44-1b3fc83d0b46","Type":"ContainerStarted","Data":"65987f47423e70fe3aad80b6d3c17d59a743847f07b486cffbfe9334ed64a7ea"}
Oct 31 23:36:51.480993 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:51.480898    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl" podStartSLOduration=2.480884094 podStartE2EDuration="2.480884094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:36:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:36:51.480315363 +0000 UTC m=+2830.227504867" watchObservedRunningTime="2026-10-31 23:36:51.480884094 +0000 UTC m=+2830.228073599"
Oct 31 23:36:52.115094 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:52.115055    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"
Oct 31 23:36:52.116872 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:52.116841    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-s6nvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 31 23:36:52.117039 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:52.116897    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl" podUID="99873d5d-87c2-423a-9a44-1b3fc83d0b46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 31 23:36:53.115129 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:53.115091    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-s6nvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 31 23:36:53.115558 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:53.115147    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl" podUID="99873d5d-87c2-423a-9a44-1b3fc83d0b46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 31 23:36:54.115572 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:54.115488    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-s6nvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 31 23:36:54.115572 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:54.115550    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl" podUID="99873d5d-87c2-423a-9a44-1b3fc83d0b46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 31 23:36:55.114738 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:55.114703    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-s6nvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 31 23:36:55.114916 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:55.114756    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl" podUID="99873d5d-87c2-423a-9a44-1b3fc83d0b46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 31 23:36:56.115228 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:56.115197    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-s6nvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 31 23:36:56.115665 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:56.115251    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl" podUID="99873d5d-87c2-423a-9a44-1b3fc83d0b46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 31 23:36:57.115064 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:57.115027    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-s6nvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 31 23:36:57.115251 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:57.115080    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl" podUID="99873d5d-87c2-423a-9a44-1b3fc83d0b46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 31 23:36:58.496089 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:58.496059    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"
Oct 31 23:36:58.496542 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:58.496342    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"
Oct 31 23:36:58.497413 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:58.497384    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl"
Oct 31 23:36:59.343176 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:59.343142    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv"
Oct 31 23:36:59.355325 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:59.355302    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv"
Oct 31 23:36:59.504767 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:36:59.504736    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv"
Oct 31 23:37:02.544758 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:37:02.544723    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:37:07.900052 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:07.900019    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:37:30.918267 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:30.918231    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:37:32.549739 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:37:32.549697    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:37:35.056739 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:35.056698    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv"]
Oct 31 23:37:40.370939 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:40.370898    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c4619f-6dbc6c67dp8772"]
Oct 31 23:37:40.395268 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:40.395201    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4619f-77c85qzrml"]
Oct 31 23:37:42.794993 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:42.794957    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:37:45.371450 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:45.371378    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4619f-6dbc6c67dp8772" podUID="64867f96-960c-4b1f-9d35-88057c41f66a" containerName="payload-processing" containerID="cri-o://2eedec1113f35e3169f1a2daa9a316a7470b430669219c68f01e96da3dc97d25" gracePeriod=25
Oct 31 23:37:45.395996 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:45.395945    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4619f-77c85qzrml" podUID="1bfe4539-8946-4dfd-9ad7-cc0937c11139" containerName="payload-pre-processing" containerID="cri-o://3a1da3aa72c2977e4ba5a01fd6395279a8d4ef8e1cd1b617a5f8a2da96b455bb" gracePeriod=25
Oct 31 23:37:50.075814 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:50.075753    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv" podUID="1f1ac3f1-e2c9-4e5e-a25a-496af40f046d" containerName="main" containerID="cri-o://47be8ace0f7ba239962d48e7d97bfff213bccd20da8d4d03fe33ea99c20b2c72" gracePeriod=45
Oct 31 23:37:50.318468 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:50.318440    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv"
Oct 31 23:37:50.357906 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:50.357819    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-tls-certs\") pod \"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d\" (UID: \"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d\") "
Oct 31 23:37:50.357906 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:50.357889    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-kube-api-access-qvlbn\" (UniqueName: \"kubernetes.io/projected/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-kube-api-access-qvlbn\") pod \"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d\" (UID: \"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d\") "
Oct 31 23:37:50.358125 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:50.357939    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-model-cache\") pod \"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d\" (UID: \"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d\") "
Oct 31 23:37:50.358125 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:50.357982    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-dshm\") pod \"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d\" (UID: \"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d\") "
Oct 31 23:37:50.358125 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:50.358030    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-tmp-dir\") pod \"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d\" (UID: \"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d\") "
Oct 31 23:37:50.358125 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:50.358092    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-home\" (UniqueName: \"kubernetes.io/empty-dir/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-home\") pod \"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d\" (UID: \"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d\") "
Oct 31 23:37:50.358373 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:50.358211    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-model-cache" pod "1f1ac3f1-e2c9-4e5e-a25a-496af40f046d" (UID: "1f1ac3f1-e2c9-4e5e-a25a-496af40f046d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:37:50.358484 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:50.358459    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-tmp-dir" pod "1f1ac3f1-e2c9-4e5e-a25a-496af40f046d" (UID: "1f1ac3f1-e2c9-4e5e-a25a-496af40f046d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:37:50.358484 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:50.358474    4301 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-model-cache\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:37:50.358603 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:50.358575    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-home" pod "1f1ac3f1-e2c9-4e5e-a25a-496af40f046d" (UID: "1f1ac3f1-e2c9-4e5e-a25a-496af40f046d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:37:50.360229 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:50.360201    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-tls-certs" pod "1f1ac3f1-e2c9-4e5e-a25a-496af40f046d" (UID: "1f1ac3f1-e2c9-4e5e-a25a-496af40f046d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:37:50.360370 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:50.360230    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-dshm" pod "1f1ac3f1-e2c9-4e5e-a25a-496af40f046d" (UID: "1f1ac3f1-e2c9-4e5e-a25a-496af40f046d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:37:50.360749 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:50.360724    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-kube-api-access-qvlbn" pod "1f1ac3f1-e2c9-4e5e-a25a-496af40f046d" (UID: "1f1ac3f1-e2c9-4e5e-a25a-496af40f046d"). InnerVolumeSpecName "kube-api-access-qvlbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:37:50.459237 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:50.459189    4301 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-tls-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:37:50.459237 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:50.459231    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvlbn\" (UniqueName: \"kubernetes.io/projected/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-kube-api-access-qvlbn\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:37:50.459237 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:50.459242    4301 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-dshm\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:37:50.459508 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:50.459268    4301 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-tmp-dir\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:37:50.459508 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:50.459276    4301 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d-home\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:37:50.701863 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:50.701828    4301 generic.go:356] "Generic (PLEG): container finished" podID="1f1ac3f1-e2c9-4e5e-a25a-496af40f046d" containerID="47be8ace0f7ba239962d48e7d97bfff213bccd20da8d4d03fe33ea99c20b2c72" exitCode=0
Oct 31 23:37:50.702068 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:50.701873    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv"
Oct 31 23:37:50.702068 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:50.701914    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv" event={"ID":"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d","Type":"ContainerDied","Data":"47be8ace0f7ba239962d48e7d97bfff213bccd20da8d4d03fe33ea99c20b2c72"}
Oct 31 23:37:50.702068 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:50.701959    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv" event={"ID":"1f1ac3f1-e2c9-4e5e-a25a-496af40f046d","Type":"ContainerDied","Data":"3571b78c167ae86732757c95075b2b0d174f8b432e123bd885abe86d79612601"}
Oct 31 23:37:50.702068 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:50.701984    4301 scope.go:122] "RemoveContainer" containerID="47be8ace0f7ba239962d48e7d97bfff213bccd20da8d4d03fe33ea99c20b2c72"
Oct 31 23:37:50.710529 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:50.710510    4301 scope.go:122] "RemoveContainer" containerID="47be8ace0f7ba239962d48e7d97bfff213bccd20da8d4d03fe33ea99c20b2c72"
Oct 31 23:37:50.710779 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:37:50.710760    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47be8ace0f7ba239962d48e7d97bfff213bccd20da8d4d03fe33ea99c20b2c72\": container with ID starting with 47be8ace0f7ba239962d48e7d97bfff213bccd20da8d4d03fe33ea99c20b2c72 not found: ID does not exist" containerID="47be8ace0f7ba239962d48e7d97bfff213bccd20da8d4d03fe33ea99c20b2c72"
Oct 31 23:37:50.710828 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:50.710787    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47be8ace0f7ba239962d48e7d97bfff213bccd20da8d4d03fe33ea99c20b2c72"} err="failed to get container status \"47be8ace0f7ba239962d48e7d97bfff213bccd20da8d4d03fe33ea99c20b2c72\": rpc error: code = NotFound desc = could not find container \"47be8ace0f7ba239962d48e7d97bfff213bccd20da8d4d03fe33ea99c20b2c72\": container with ID starting with 47be8ace0f7ba239962d48e7d97bfff213bccd20da8d4d03fe33ea99c20b2c72 not found: ID does not exist"
Oct 31 23:37:50.721255 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:50.721232    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv"]
Oct 31 23:37:50.725551 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:50.725528    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-c4619f/simulator-single-main-c4619f-kserve-6f99b897cf-k7ldv"]
Oct 31 23:37:51.401852 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:51.401822    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f1ac3f1-e2c9-4e5e-a25a-496af40f046d" path="/var/lib/kubelet/pods/1f1ac3f1-e2c9-4e5e-a25a-496af40f046d/volumes"
Oct 31 23:37:55.373816 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:55.373778    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4619f-6dbc6c67dp8772" podUID="64867f96-960c-4b1f-9d35-88057c41f66a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 23:37:55.374205 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:55.373780    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4619f-77c85qzrml" podUID="1bfe4539-8946-4dfd-9ad7-cc0937c11139" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 23:37:59.982934 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:37:59.982901    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:38:02.545902 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:38:02.545862    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:38:03.411897 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:03.411863    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:38:05.374516 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:05.374482    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4619f-77c85qzrml" podUID="1bfe4539-8946-4dfd-9ad7-cc0937c11139" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 23:38:05.374900 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:05.374496    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4619f-6dbc6c67dp8772" podUID="64867f96-960c-4b1f-9d35-88057c41f66a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 23:38:05.652318 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:05.652292    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4619f-6dbc6c67dp8772"
Oct 31 23:38:05.654807 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:05.654789    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4619f-77c85qzrml"
Oct 31 23:38:05.765189 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:05.765159    4301 generic.go:356] "Generic (PLEG): container finished" podID="1bfe4539-8946-4dfd-9ad7-cc0937c11139" containerID="3a1da3aa72c2977e4ba5a01fd6395279a8d4ef8e1cd1b617a5f8a2da96b455bb" exitCode=0
Oct 31 23:38:05.765433 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:05.765206    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4619f-77c85qzrml"
Oct 31 23:38:05.765433 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:05.765236    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4619f-77c85qzrml" event={"ID":"1bfe4539-8946-4dfd-9ad7-cc0937c11139","Type":"ContainerDied","Data":"3a1da3aa72c2977e4ba5a01fd6395279a8d4ef8e1cd1b617a5f8a2da96b455bb"}
Oct 31 23:38:05.765433 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:05.765297    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4619f-77c85qzrml" event={"ID":"1bfe4539-8946-4dfd-9ad7-cc0937c11139","Type":"ContainerDied","Data":"884a6e153348861ba380f1dd6465d941fc7a7bd95fa2630fbb7299d149b628b1"}
Oct 31 23:38:05.765433 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:05.765328    4301 scope.go:122] "RemoveContainer" containerID="3a1da3aa72c2977e4ba5a01fd6395279a8d4ef8e1cd1b617a5f8a2da96b455bb"
Oct 31 23:38:05.766664 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:05.766644    4301 generic.go:356] "Generic (PLEG): container finished" podID="64867f96-960c-4b1f-9d35-88057c41f66a" containerID="2eedec1113f35e3169f1a2daa9a316a7470b430669219c68f01e96da3dc97d25" exitCode=0
Oct 31 23:38:05.766760 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:05.766692    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4619f-6dbc6c67dp8772" event={"ID":"64867f96-960c-4b1f-9d35-88057c41f66a","Type":"ContainerDied","Data":"2eedec1113f35e3169f1a2daa9a316a7470b430669219c68f01e96da3dc97d25"}
Oct 31 23:38:05.766760 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:05.766704    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4619f-6dbc6c67dp8772"
Oct 31 23:38:05.766760 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:05.766720    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4619f-6dbc6c67dp8772" event={"ID":"64867f96-960c-4b1f-9d35-88057c41f66a","Type":"ContainerDied","Data":"b5983533c9125e31ed8a9ea57375ae5c9c340b0fde8b1a437deaeb1b9e6cad22"}
Oct 31 23:38:05.773304 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:05.773288    4301 scope.go:122] "RemoveContainer" containerID="3a1da3aa72c2977e4ba5a01fd6395279a8d4ef8e1cd1b617a5f8a2da96b455bb"
Oct 31 23:38:05.773568 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:38:05.773549    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a1da3aa72c2977e4ba5a01fd6395279a8d4ef8e1cd1b617a5f8a2da96b455bb\": container with ID starting with 3a1da3aa72c2977e4ba5a01fd6395279a8d4ef8e1cd1b617a5f8a2da96b455bb not found: ID does not exist" containerID="3a1da3aa72c2977e4ba5a01fd6395279a8d4ef8e1cd1b617a5f8a2da96b455bb"
Oct 31 23:38:05.773650 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:05.773574    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a1da3aa72c2977e4ba5a01fd6395279a8d4ef8e1cd1b617a5f8a2da96b455bb"} err="failed to get container status \"3a1da3aa72c2977e4ba5a01fd6395279a8d4ef8e1cd1b617a5f8a2da96b455bb\": rpc error: code = NotFound desc = could not find container \"3a1da3aa72c2977e4ba5a01fd6395279a8d4ef8e1cd1b617a5f8a2da96b455bb\": container with ID starting with 3a1da3aa72c2977e4ba5a01fd6395279a8d4ef8e1cd1b617a5f8a2da96b455bb not found: ID does not exist"
Oct 31 23:38:05.773650 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:05.773588    4301 scope.go:122] "RemoveContainer" containerID="2eedec1113f35e3169f1a2daa9a316a7470b430669219c68f01e96da3dc97d25"
Oct 31 23:38:05.780887 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:05.780871    4301 scope.go:122] "RemoveContainer" containerID="2eedec1113f35e3169f1a2daa9a316a7470b430669219c68f01e96da3dc97d25"
Oct 31 23:38:05.781130 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:38:05.781112    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eedec1113f35e3169f1a2daa9a316a7470b430669219c68f01e96da3dc97d25\": container with ID starting with 2eedec1113f35e3169f1a2daa9a316a7470b430669219c68f01e96da3dc97d25 not found: ID does not exist" containerID="2eedec1113f35e3169f1a2daa9a316a7470b430669219c68f01e96da3dc97d25"
Oct 31 23:38:05.781170 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:05.781142    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eedec1113f35e3169f1a2daa9a316a7470b430669219c68f01e96da3dc97d25"} err="failed to get container status \"2eedec1113f35e3169f1a2daa9a316a7470b430669219c68f01e96da3dc97d25\": rpc error: code = NotFound desc = could not find container \"2eedec1113f35e3169f1a2daa9a316a7470b430669219c68f01e96da3dc97d25\": container with ID starting with 2eedec1113f35e3169f1a2daa9a316a7470b430669219c68f01e96da3dc97d25 not found: ID does not exist"
Oct 31 23:38:05.786449 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:05.786433    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64867f96-960c-4b1f-9d35-88057c41f66a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64867f96-960c-4b1f-9d35-88057c41f66a-plugins-config-volume\") pod \"64867f96-960c-4b1f-9d35-88057c41f66a\" (UID: \"64867f96-960c-4b1f-9d35-88057c41f66a\") "
Oct 31 23:38:05.786502 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:05.786477    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bfe4539-8946-4dfd-9ad7-cc0937c11139-kube-api-access-vtgkb\" (UniqueName: \"kubernetes.io/projected/1bfe4539-8946-4dfd-9ad7-cc0937c11139-kube-api-access-vtgkb\") pod \"1bfe4539-8946-4dfd-9ad7-cc0937c11139\" (UID: \"1bfe4539-8946-4dfd-9ad7-cc0937c11139\") "
Oct 31 23:38:05.786543 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:05.786516    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64867f96-960c-4b1f-9d35-88057c41f66a-kube-api-access-xn6bn\" (UniqueName: \"kubernetes.io/projected/64867f96-960c-4b1f-9d35-88057c41f66a-kube-api-access-xn6bn\") pod \"64867f96-960c-4b1f-9d35-88057c41f66a\" (UID: \"64867f96-960c-4b1f-9d35-88057c41f66a\") "
Oct 31 23:38:05.786543 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:05.786539    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bfe4539-8946-4dfd-9ad7-cc0937c11139-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bfe4539-8946-4dfd-9ad7-cc0937c11139-plugins-config-volume\") pod \"1bfe4539-8946-4dfd-9ad7-cc0937c11139\" (UID: \"1bfe4539-8946-4dfd-9ad7-cc0937c11139\") "
Oct 31 23:38:05.786947 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:05.786798    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64867f96-960c-4b1f-9d35-88057c41f66a-plugins-config-volume" pod "64867f96-960c-4b1f-9d35-88057c41f66a" (UID: "64867f96-960c-4b1f-9d35-88057c41f66a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:38:05.787053 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:05.786966    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bfe4539-8946-4dfd-9ad7-cc0937c11139-plugins-config-volume" pod "1bfe4539-8946-4dfd-9ad7-cc0937c11139" (UID: "1bfe4539-8946-4dfd-9ad7-cc0937c11139"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:38:05.788521 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:05.788503    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bfe4539-8946-4dfd-9ad7-cc0937c11139-kube-api-access-vtgkb" pod "1bfe4539-8946-4dfd-9ad7-cc0937c11139" (UID: "1bfe4539-8946-4dfd-9ad7-cc0937c11139"). InnerVolumeSpecName "kube-api-access-vtgkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:38:05.788601 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:05.788555    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64867f96-960c-4b1f-9d35-88057c41f66a-kube-api-access-xn6bn" pod "64867f96-960c-4b1f-9d35-88057c41f66a" (UID: "64867f96-960c-4b1f-9d35-88057c41f66a"). InnerVolumeSpecName "kube-api-access-xn6bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:38:05.887817 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:05.887745    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtgkb\" (UniqueName: \"kubernetes.io/projected/1bfe4539-8946-4dfd-9ad7-cc0937c11139-kube-api-access-vtgkb\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:38:05.887817 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:05.887771    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn6bn\" (UniqueName: \"kubernetes.io/projected/64867f96-960c-4b1f-9d35-88057c41f66a-kube-api-access-xn6bn\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:38:05.887817 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:05.887779    4301 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bfe4539-8946-4dfd-9ad7-cc0937c11139-plugins-config-volume\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:38:05.887817 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:05.887790    4301 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64867f96-960c-4b1f-9d35-88057c41f66a-plugins-config-volume\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:38:06.088301 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:06.088274    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4619f-77c85qzrml"]
Oct 31 23:38:06.092668 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:06.092643    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4619f-77c85qzrml"]
Oct 31 23:38:06.102631 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:06.102609    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c4619f-6dbc6c67dp8772"]
Oct 31 23:38:06.106755 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:06.106733    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c4619f-6dbc6c67dp8772"]
Oct 31 23:38:07.401837 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:07.401805    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bfe4539-8946-4dfd-9ad7-cc0937c11139" path="/var/lib/kubelet/pods/1bfe4539-8946-4dfd-9ad7-cc0937c11139/volumes"
Oct 31 23:38:07.402225 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:07.402148    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64867f96-960c-4b1f-9d35-88057c41f66a" path="/var/lib/kubelet/pods/64867f96-960c-4b1f-9d35-88057c41f66a/volumes"
Oct 31 23:38:32.545601 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:38:32.545516    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:38:34.810860 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.810823    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"]
Oct 31 23:38:34.811420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.811157    4301 cpu_manager.go:414] "containerMap: removing container" podUID="1bfe4539-8946-4dfd-9ad7-cc0937c11139" containerName="payload-pre-processing"
Oct 31 23:38:34.811420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.811170    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bfe4539-8946-4dfd-9ad7-cc0937c11139" containerName="payload-pre-processing"
Oct 31 23:38:34.811420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.811189    4301 cpu_manager.go:414] "containerMap: removing container" podUID="64867f96-960c-4b1f-9d35-88057c41f66a" containerName="payload-processing"
Oct 31 23:38:34.811420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.811194    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64867f96-960c-4b1f-9d35-88057c41f66a" containerName="payload-processing"
Oct 31 23:38:34.811420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.811203    4301 cpu_manager.go:414] "containerMap: removing container" podUID="1f1ac3f1-e2c9-4e5e-a25a-496af40f046d" containerName="main"
Oct 31 23:38:34.811420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.811208    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f1ac3f1-e2c9-4e5e-a25a-496af40f046d" containerName="main"
Oct 31 23:38:34.811420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.811263    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="64867f96-960c-4b1f-9d35-88057c41f66a" containerName="payload-processing"
Oct 31 23:38:34.811420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.811271    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f1ac3f1-e2c9-4e5e-a25a-496af40f046d" containerName="main"
Oct 31 23:38:34.811420 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.811282    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bfe4539-8946-4dfd-9ad7-cc0937c11139" containerName="payload-pre-processing"
Oct 31 23:38:34.814443 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.814411    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:38:34.833859 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.833813    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"]
Oct 31 23:38:34.842138 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.842108    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt7vq\" (UniqueName: \"kubernetes.io/projected/e6602036-1c07-4882-ab1a-1dbbd0a857f5-kube-api-access-rt7vq\") pod \"maas-default-gateway-openshift-default-6886b96f9-6qscl\" (UID: \"e6602036-1c07-4882-ab1a-1dbbd0a857f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:38:34.842285 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.842161    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6602036-1c07-4882-ab1a-1dbbd0a857f5-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6qscl\" (UID: \"e6602036-1c07-4882-ab1a-1dbbd0a857f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:38:34.842285 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.842242    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6602036-1c07-4882-ab1a-1dbbd0a857f5-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-6qscl\" (UID: \"e6602036-1c07-4882-ab1a-1dbbd0a857f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:38:34.842430 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.842294    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6602036-1c07-4882-ab1a-1dbbd0a857f5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-6qscl\" (UID: \"e6602036-1c07-4882-ab1a-1dbbd0a857f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:38:34.842430 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.842313    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6602036-1c07-4882-ab1a-1dbbd0a857f5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-6qscl\" (UID: \"e6602036-1c07-4882-ab1a-1dbbd0a857f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:38:34.842430 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.842352    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6602036-1c07-4882-ab1a-1dbbd0a857f5-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-6qscl\" (UID: \"e6602036-1c07-4882-ab1a-1dbbd0a857f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:38:34.842430 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.842403    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6602036-1c07-4882-ab1a-1dbbd0a857f5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-6qscl\" (UID: \"e6602036-1c07-4882-ab1a-1dbbd0a857f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:38:34.842430 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.842420    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6602036-1c07-4882-ab1a-1dbbd0a857f5-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-6qscl\" (UID: \"e6602036-1c07-4882-ab1a-1dbbd0a857f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:38:34.842708 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.842536    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6602036-1c07-4882-ab1a-1dbbd0a857f5-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6qscl\" (UID: \"e6602036-1c07-4882-ab1a-1dbbd0a857f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:38:34.943161 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.943130    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6602036-1c07-4882-ab1a-1dbbd0a857f5-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-6qscl\" (UID: \"e6602036-1c07-4882-ab1a-1dbbd0a857f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:38:34.943161 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.943167    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6602036-1c07-4882-ab1a-1dbbd0a857f5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-6qscl\" (UID: \"e6602036-1c07-4882-ab1a-1dbbd0a857f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:38:34.943425 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.943193    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6602036-1c07-4882-ab1a-1dbbd0a857f5-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-6qscl\" (UID: \"e6602036-1c07-4882-ab1a-1dbbd0a857f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:38:34.943425 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.943225    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6602036-1c07-4882-ab1a-1dbbd0a857f5-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6qscl\" (UID: \"e6602036-1c07-4882-ab1a-1dbbd0a857f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:38:34.943425 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.943257    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt7vq\" (UniqueName: \"kubernetes.io/projected/e6602036-1c07-4882-ab1a-1dbbd0a857f5-kube-api-access-rt7vq\") pod \"maas-default-gateway-openshift-default-6886b96f9-6qscl\" (UID: \"e6602036-1c07-4882-ab1a-1dbbd0a857f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:38:34.943425 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.943292    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6602036-1c07-4882-ab1a-1dbbd0a857f5-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6qscl\" (UID: \"e6602036-1c07-4882-ab1a-1dbbd0a857f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:38:34.943425 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.943315    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6602036-1c07-4882-ab1a-1dbbd0a857f5-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-6qscl\" (UID: \"e6602036-1c07-4882-ab1a-1dbbd0a857f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:38:34.943425 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.943341    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6602036-1c07-4882-ab1a-1dbbd0a857f5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-6qscl\" (UID: \"e6602036-1c07-4882-ab1a-1dbbd0a857f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:38:34.943425 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.943386    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6602036-1c07-4882-ab1a-1dbbd0a857f5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-6qscl\" (UID: \"e6602036-1c07-4882-ab1a-1dbbd0a857f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:38:34.943878 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.943692    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6602036-1c07-4882-ab1a-1dbbd0a857f5-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6qscl\" (UID: \"e6602036-1c07-4882-ab1a-1dbbd0a857f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:38:34.943878 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.943751    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6602036-1c07-4882-ab1a-1dbbd0a857f5-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-6qscl\" (UID: \"e6602036-1c07-4882-ab1a-1dbbd0a857f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:38:34.943878 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.943795    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6602036-1c07-4882-ab1a-1dbbd0a857f5-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-6qscl\" (UID: \"e6602036-1c07-4882-ab1a-1dbbd0a857f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:38:34.943998 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.943933    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6602036-1c07-4882-ab1a-1dbbd0a857f5-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6qscl\" (UID: \"e6602036-1c07-4882-ab1a-1dbbd0a857f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:38:34.943998 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.943941    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6602036-1c07-4882-ab1a-1dbbd0a857f5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-6qscl\" (UID: \"e6602036-1c07-4882-ab1a-1dbbd0a857f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:38:34.945794 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.945773    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6602036-1c07-4882-ab1a-1dbbd0a857f5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-6qscl\" (UID: \"e6602036-1c07-4882-ab1a-1dbbd0a857f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:38:34.945972 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.945953    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6602036-1c07-4882-ab1a-1dbbd0a857f5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-6qscl\" (UID: \"e6602036-1c07-4882-ab1a-1dbbd0a857f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:38:34.954729 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.954707    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6602036-1c07-4882-ab1a-1dbbd0a857f5-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-6qscl\" (UID: \"e6602036-1c07-4882-ab1a-1dbbd0a857f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:38:34.954876 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:34.954858    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt7vq\" (UniqueName: \"kubernetes.io/projected/e6602036-1c07-4882-ab1a-1dbbd0a857f5-kube-api-access-rt7vq\") pod \"maas-default-gateway-openshift-default-6886b96f9-6qscl\" (UID: \"e6602036-1c07-4882-ab1a-1dbbd0a857f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:38:35.127864 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:35.127830    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:38:35.407590 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:35.407505    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"]
Oct 31 23:38:35.410763 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:38:35.410742    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6602036_1c07_4882_ab1a_1dbbd0a857f5.slice/crio-6b594c1c0e2e97f0e4286cebc60f903899852a879161782e2eafbd9bdd59e706 WatchSource:0}: Error finding container 6b594c1c0e2e97f0e4286cebc60f903899852a879161782e2eafbd9bdd59e706: Status 404 returned error can't find the container with id 6b594c1c0e2e97f0e4286cebc60f903899852a879161782e2eafbd9bdd59e706
Oct 31 23:38:35.413005 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:35.412973    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:38:35.413069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:35.413039    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:38:35.413110 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:35.413066    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:38:35.886287 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:35.886246    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl" event={"ID":"e6602036-1c07-4882-ab1a-1dbbd0a857f5","Type":"ContainerStarted","Data":"899cd63704bb26d8b2dc480ff8dee37ef65a3c1e7f2ebb98c1aec1c7dfd9679c"}
Oct 31 23:38:35.886692 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:35.886310    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl" event={"ID":"e6602036-1c07-4882-ab1a-1dbbd0a857f5","Type":"ContainerStarted","Data":"6b594c1c0e2e97f0e4286cebc60f903899852a879161782e2eafbd9bdd59e706"}
Oct 31 23:38:35.908399 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:35.908336    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl" podStartSLOduration=1.908324562 podStartE2EDuration="1.908324562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:38:35.907503295 +0000 UTC m=+2934.654692798" watchObservedRunningTime="2026-10-31 23:38:35.908324562 +0000 UTC m=+2934.655514066"
Oct 31 23:38:36.128998 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:36.128965    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:38:36.130474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:36.130450    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6qscl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 31 23:38:36.130603 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:36.130494    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl" podUID="e6602036-1c07-4882-ab1a-1dbbd0a857f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 31 23:38:37.132806 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:37.132780    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:38:37.894272 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:37.894239    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:38:37.895341 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:37.895320    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:38:49.818678 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:49.818644    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"]
Oct 31 23:38:49.824302 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:49.824279    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:38:49.839200 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:49.839174    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"]
Oct 31 23:38:49.877620 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:49.877588    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea56eff-0929-4f33-9adc-f10042fb1b96-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9tfc\" (UID: \"8ea56eff-0929-4f33-9adc-f10042fb1b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:38:49.877792 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:49.877651    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea56eff-0929-4f33-9adc-f10042fb1b96-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9tfc\" (UID: \"8ea56eff-0929-4f33-9adc-f10042fb1b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:38:49.877792 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:49.877745    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26rh5\" (UniqueName: \"kubernetes.io/projected/8ea56eff-0929-4f33-9adc-f10042fb1b96-kube-api-access-26rh5\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9tfc\" (UID: \"8ea56eff-0929-4f33-9adc-f10042fb1b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:38:49.877792 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:49.877779    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea56eff-0929-4f33-9adc-f10042fb1b96-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9tfc\" (UID: \"8ea56eff-0929-4f33-9adc-f10042fb1b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:38:49.877923 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:49.877810    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea56eff-0929-4f33-9adc-f10042fb1b96-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9tfc\" (UID: \"8ea56eff-0929-4f33-9adc-f10042fb1b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:38:49.877923 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:49.877903    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea56eff-0929-4f33-9adc-f10042fb1b96-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9tfc\" (UID: \"8ea56eff-0929-4f33-9adc-f10042fb1b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:38:49.878011 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:49.877942    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea56eff-0929-4f33-9adc-f10042fb1b96-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9tfc\" (UID: \"8ea56eff-0929-4f33-9adc-f10042fb1b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:38:49.878011 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:49.877961    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea56eff-0929-4f33-9adc-f10042fb1b96-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9tfc\" (UID: \"8ea56eff-0929-4f33-9adc-f10042fb1b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:38:49.878011 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:49.877978    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea56eff-0929-4f33-9adc-f10042fb1b96-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9tfc\" (UID: \"8ea56eff-0929-4f33-9adc-f10042fb1b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:38:49.978457 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:49.978425    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea56eff-0929-4f33-9adc-f10042fb1b96-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9tfc\" (UID: \"8ea56eff-0929-4f33-9adc-f10042fb1b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:38:49.978664 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:49.978468    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea56eff-0929-4f33-9adc-f10042fb1b96-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9tfc\" (UID: \"8ea56eff-0929-4f33-9adc-f10042fb1b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:38:49.978664 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:49.978587    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea56eff-0929-4f33-9adc-f10042fb1b96-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9tfc\" (UID: \"8ea56eff-0929-4f33-9adc-f10042fb1b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:38:49.978664 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:49.978617    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26rh5\" (UniqueName: \"kubernetes.io/projected/8ea56eff-0929-4f33-9adc-f10042fb1b96-kube-api-access-26rh5\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9tfc\" (UID: \"8ea56eff-0929-4f33-9adc-f10042fb1b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:38:49.978664 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:49.978638    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea56eff-0929-4f33-9adc-f10042fb1b96-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9tfc\" (UID: \"8ea56eff-0929-4f33-9adc-f10042fb1b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:38:49.978664 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:49.978664    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea56eff-0929-4f33-9adc-f10042fb1b96-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9tfc\" (UID: \"8ea56eff-0929-4f33-9adc-f10042fb1b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:38:49.978947 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:49.978849    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea56eff-0929-4f33-9adc-f10042fb1b96-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9tfc\" (UID: \"8ea56eff-0929-4f33-9adc-f10042fb1b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:38:49.978947 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:49.978893    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea56eff-0929-4f33-9adc-f10042fb1b96-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9tfc\" (UID: \"8ea56eff-0929-4f33-9adc-f10042fb1b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:38:49.978947 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:49.978903    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea56eff-0929-4f33-9adc-f10042fb1b96-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9tfc\" (UID: \"8ea56eff-0929-4f33-9adc-f10042fb1b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:38:49.979123 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:49.978987    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea56eff-0929-4f33-9adc-f10042fb1b96-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9tfc\" (UID: \"8ea56eff-0929-4f33-9adc-f10042fb1b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:38:49.979123 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:49.978986    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea56eff-0929-4f33-9adc-f10042fb1b96-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9tfc\" (UID: \"8ea56eff-0929-4f33-9adc-f10042fb1b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:38:49.979123 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:49.979088    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea56eff-0929-4f33-9adc-f10042fb1b96-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9tfc\" (UID: \"8ea56eff-0929-4f33-9adc-f10042fb1b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:38:49.979232 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:49.979144    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea56eff-0929-4f33-9adc-f10042fb1b96-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9tfc\" (UID: \"8ea56eff-0929-4f33-9adc-f10042fb1b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:38:49.979518 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:49.979501    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea56eff-0929-4f33-9adc-f10042fb1b96-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9tfc\" (UID: \"8ea56eff-0929-4f33-9adc-f10042fb1b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:38:49.980950 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:49.980931    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea56eff-0929-4f33-9adc-f10042fb1b96-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9tfc\" (UID: \"8ea56eff-0929-4f33-9adc-f10042fb1b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:38:49.981320 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:49.981299    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea56eff-0929-4f33-9adc-f10042fb1b96-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9tfc\" (UID: \"8ea56eff-0929-4f33-9adc-f10042fb1b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:38:49.991120 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:49.991097    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea56eff-0929-4f33-9adc-f10042fb1b96-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9tfc\" (UID: \"8ea56eff-0929-4f33-9adc-f10042fb1b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:38:49.991492 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:49.991454    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26rh5\" (UniqueName: \"kubernetes.io/projected/8ea56eff-0929-4f33-9adc-f10042fb1b96-kube-api-access-26rh5\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9tfc\" (UID: \"8ea56eff-0929-4f33-9adc-f10042fb1b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:38:50.138445 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:50.138409    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:38:50.419421 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:50.419308    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"]
Oct 31 23:38:50.423331 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:38:50.423303    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea56eff_0929_4f33_9adc_f10042fb1b96.slice/crio-526630659e0ac04c735f886a869b1626de8d705eeddf980853d802a0f67533b3 WatchSource:0}: Error finding container 526630659e0ac04c735f886a869b1626de8d705eeddf980853d802a0f67533b3: Status 404 returned error can't find the container with id 526630659e0ac04c735f886a869b1626de8d705eeddf980853d802a0f67533b3
Oct 31 23:38:50.425390 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:50.425335    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:38:50.425476 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:50.425431    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:38:50.425476 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:50.425465    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:38:50.945501 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:50.945462    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc" event={"ID":"8ea56eff-0929-4f33-9adc-f10042fb1b96","Type":"ContainerStarted","Data":"f4f291bb9fdcdae56559840ca8f24e2a90592e8859b193aabeca37f55bad9bf1"}
Oct 31 23:38:50.945907 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:50.945515    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc" event={"ID":"8ea56eff-0929-4f33-9adc-f10042fb1b96","Type":"ContainerStarted","Data":"526630659e0ac04c735f886a869b1626de8d705eeddf980853d802a0f67533b3"}
Oct 31 23:38:50.970478 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:50.970434    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc" podStartSLOduration=1.9704204889999999 podStartE2EDuration="1.970420489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:38:50.969731716 +0000 UTC m=+2949.716921220" watchObservedRunningTime="2026-10-31 23:38:50.970420489 +0000 UTC m=+2949.717609995"
Oct 31 23:38:51.139333 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:51.139299    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:38:51.140769 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:51.140739    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-z9tfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 31 23:38:51.140911 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:51.140788    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc" podUID="8ea56eff-0929-4f33-9adc-f10042fb1b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 31 23:38:52.143135 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:52.143103    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:38:52.954891 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:52.954862    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:38:52.955873 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:38:52.955851    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:39:02.544733 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:39:02.544696    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:39:32.544648 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:39:32.544608    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:39:41.876464 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:41.876435    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s9nl_1f0042ee-6258-4aa8-af48-7568f2f2e8a1/ovn-acl-logging/0.log"
Oct 31 23:39:41.879214 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:41.879194    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s9nl_1f0042ee-6258-4aa8-af48-7568f2f2e8a1/ovn-acl-logging/0.log"
Oct 31 23:39:44.895385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:44.895342    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:39:53.936637 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:53.936538    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6qscl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 23:39:53.936637 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:53.936598    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl" podUID="e6602036-1c07-4882-ab1a-1dbbd0a857f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:39:53.996629 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:53.996586    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-z9tfc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 23:39:53.996827 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:53.996675    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc" podUID="8ea56eff-0929-4f33-9adc-f10042fb1b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:39:55.238205 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:55.238169    4301 generic.go:356] "Generic (PLEG): container finished" podID="8ea56eff-0929-4f33-9adc-f10042fb1b96" containerID="f4f291bb9fdcdae56559840ca8f24e2a90592e8859b193aabeca37f55bad9bf1" exitCode=0
Oct 31 23:39:55.238734 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:55.238243    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc" event={"ID":"8ea56eff-0929-4f33-9adc-f10042fb1b96","Type":"ContainerDied","Data":"f4f291bb9fdcdae56559840ca8f24e2a90592e8859b193aabeca37f55bad9bf1"}
Oct 31 23:39:55.238734 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:55.238672    4301 scope.go:122] "RemoveContainer" containerID="f4f291bb9fdcdae56559840ca8f24e2a90592e8859b193aabeca37f55bad9bf1"
Oct 31 23:39:55.239817 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:55.239784    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:39:55.239914 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:55.239873    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:39:55.239964 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:55.239921    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:39:55.239964 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:55.239946    4301 generic.go:356] "Generic (PLEG): container finished" podID="e6602036-1c07-4882-ab1a-1dbbd0a857f5" containerID="899cd63704bb26d8b2dc480ff8dee37ef65a3c1e7f2ebb98c1aec1c7dfd9679c" exitCode=0
Oct 31 23:39:55.240047 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:55.240016    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl" event={"ID":"e6602036-1c07-4882-ab1a-1dbbd0a857f5","Type":"ContainerDied","Data":"899cd63704bb26d8b2dc480ff8dee37ef65a3c1e7f2ebb98c1aec1c7dfd9679c"}
Oct 31 23:39:55.240332 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:55.240314    4301 scope.go:122] "RemoveContainer" containerID="899cd63704bb26d8b2dc480ff8dee37ef65a3c1e7f2ebb98c1aec1c7dfd9679c"
Oct 31 23:39:55.241337 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:55.241313    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:39:55.241428 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:55.241391    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:39:55.241490 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:55.241431    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:39:56.277753 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:56.277714    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc" event={"ID":"8ea56eff-0929-4f33-9adc-f10042fb1b96","Type":"ContainerStarted","Data":"b59db7665ac5a1622f78c059f7a46bb388c94562b0bb53b166bc7fa964304479"}
Oct 31 23:39:56.281899 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:56.281868    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl" event={"ID":"e6602036-1c07-4882-ab1a-1dbbd0a857f5","Type":"ContainerStarted","Data":"f22da23a3f03aa12abf5e6217cfc383d64187ffa32549e4d136fd6066080c65f"}
Oct 31 23:39:57.130567 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:57.130524    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:39:57.131300 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:57.131272    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6qscl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 31 23:39:57.131529 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:57.131505    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl" podUID="e6602036-1c07-4882-ab1a-1dbbd0a857f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 31 23:39:57.141919 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:57.141726    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:39:57.141919 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:57.141832    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-z9tfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 31 23:39:57.141919 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:57.141869    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc" podUID="8ea56eff-0929-4f33-9adc-f10042fb1b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 31 23:39:58.132828 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:58.132785    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6qscl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 31 23:39:58.133472 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:58.133437    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl" podUID="e6602036-1c07-4882-ab1a-1dbbd0a857f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 31 23:39:58.142746 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:58.142715    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-z9tfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 31 23:39:58.142979 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:58.142956    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc" podUID="8ea56eff-0929-4f33-9adc-f10042fb1b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 31 23:39:59.137382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:59.132827    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6qscl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 31 23:39:59.137382 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:59.132893    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl" podUID="e6602036-1c07-4882-ab1a-1dbbd0a857f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 31 23:39:59.146783 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:59.146748    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-z9tfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 31 23:39:59.147080 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:39:59.147053    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc" podUID="8ea56eff-0929-4f33-9adc-f10042fb1b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 31 23:40:00.132391 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:00.128470    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6qscl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 31 23:40:00.132391 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:00.128531    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl" podUID="e6602036-1c07-4882-ab1a-1dbbd0a857f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 31 23:40:00.159381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:00.155673    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-z9tfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 31 23:40:00.159381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:00.155742    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc" podUID="8ea56eff-0929-4f33-9adc-f10042fb1b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 31 23:40:01.132822 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:01.132780    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6qscl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 31 23:40:01.133125 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:01.133094    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl" podUID="e6602036-1c07-4882-ab1a-1dbbd0a857f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 31 23:40:01.141736 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:01.141704    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-z9tfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 31 23:40:01.142504 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:01.142474    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc" podUID="8ea56eff-0929-4f33-9adc-f10042fb1b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 31 23:40:02.128410 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:02.128340    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6qscl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 31 23:40:02.128845 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:02.128454    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl" podUID="e6602036-1c07-4882-ab1a-1dbbd0a857f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 31 23:40:02.139816 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:02.139784    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-z9tfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 31 23:40:02.139921 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:02.139840    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc" podUID="8ea56eff-0929-4f33-9adc-f10042fb1b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 31 23:40:02.546006 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:40:02.545917    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:40:03.438341 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:03.438294    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:40:03.439010 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:03.438988    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:40:03.439911 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:03.439885    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9tfc"
Oct 31 23:40:03.464790 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:03.464766    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:40:03.465139 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:03.465115    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:40:03.465966 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:03.465945    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6qscl"
Oct 31 23:40:09.611694 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:09.611659    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:40:13.967383 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:13.960033    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:40:14.502286 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:14.502237    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-s6nvl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 23:40:14.502685 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:14.502642    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl" podUID="99873d5d-87c2-423a-9a44-1b3fc83d0b46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:40:28.626582 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:28.626544    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:40:32.544964 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:40:32.544928    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:40:34.936127 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:34.936089    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5"]
Oct 31 23:40:34.948806 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:34.948777    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5"
Oct 31 23:40:34.951352 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:34.951322    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5"]
Oct 31 23:40:34.984368 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:34.984323    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-abd9b5-serving-cert\""
Oct 31 23:40:35.065651 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:35.065596    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/233555e0-e0b6-4610-ae19-675eb13b7a78-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5\" (UID: \"233555e0-e0b6-4610-ae19-675eb13b7a78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5"
Oct 31 23:40:35.065847 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:35.065723    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/233555e0-e0b6-4610-ae19-675eb13b7a78-metrics-certs\") pod \"maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5\" (UID: \"233555e0-e0b6-4610-ae19-675eb13b7a78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5"
Oct 31 23:40:35.065908 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:35.065869    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxtq9\" (UniqueName: \"kubernetes.io/projected/233555e0-e0b6-4610-ae19-675eb13b7a78-kube-api-access-dxtq9\") pod \"maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5\" (UID: \"233555e0-e0b6-4610-ae19-675eb13b7a78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5"
Oct 31 23:40:35.065947 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:35.065901    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/233555e0-e0b6-4610-ae19-675eb13b7a78-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5\" (UID: \"233555e0-e0b6-4610-ae19-675eb13b7a78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5"
Oct 31 23:40:35.166552 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:35.166515    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxtq9\" (UniqueName: \"kubernetes.io/projected/233555e0-e0b6-4610-ae19-675eb13b7a78-kube-api-access-dxtq9\") pod \"maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5\" (UID: \"233555e0-e0b6-4610-ae19-675eb13b7a78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5"
Oct 31 23:40:35.166552 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:35.166550    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/233555e0-e0b6-4610-ae19-675eb13b7a78-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5\" (UID: \"233555e0-e0b6-4610-ae19-675eb13b7a78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5"
Oct 31 23:40:35.166804 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:35.166593    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/233555e0-e0b6-4610-ae19-675eb13b7a78-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5\" (UID: \"233555e0-e0b6-4610-ae19-675eb13b7a78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5"
Oct 31 23:40:35.166804 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:35.166730    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/233555e0-e0b6-4610-ae19-675eb13b7a78-metrics-certs\") pod \"maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5\" (UID: \"233555e0-e0b6-4610-ae19-675eb13b7a78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5"
Oct 31 23:40:35.167161 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:35.167141    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/233555e0-e0b6-4610-ae19-675eb13b7a78-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5\" (UID: \"233555e0-e0b6-4610-ae19-675eb13b7a78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5"
Oct 31 23:40:35.169036 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:35.169010    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/233555e0-e0b6-4610-ae19-675eb13b7a78-metrics-certs\") pod \"maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5\" (UID: \"233555e0-e0b6-4610-ae19-675eb13b7a78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5"
Oct 31 23:40:35.169194 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:35.169174    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/233555e0-e0b6-4610-ae19-675eb13b7a78-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5\" (UID: \"233555e0-e0b6-4610-ae19-675eb13b7a78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5"
Oct 31 23:40:35.179217 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:35.179193    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxtq9\" (UniqueName: \"kubernetes.io/projected/233555e0-e0b6-4610-ae19-675eb13b7a78-kube-api-access-dxtq9\") pod \"maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5\" (UID: \"233555e0-e0b6-4610-ae19-675eb13b7a78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5"
Oct 31 23:40:35.265032 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:35.264946    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5"
Oct 31 23:40:35.541767 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:35.541713    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5"]
Oct 31 23:40:35.544177 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:40:35.544149    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod233555e0_e0b6_4610_ae19_675eb13b7a78.slice/crio-6d0ed3fc61262af40382ff7511dd00534b8f334599b0913d8f92e1b4b9ed7d01 WatchSource:0}: Error finding container 6d0ed3fc61262af40382ff7511dd00534b8f334599b0913d8f92e1b4b9ed7d01: Status 404 returned error can't find the container with id 6d0ed3fc61262af40382ff7511dd00534b8f334599b0913d8f92e1b4b9ed7d01
Oct 31 23:40:35.545988 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:35.545970    4301 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 23:40:36.491414 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:36.491381    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5" event={"ID":"233555e0-e0b6-4610-ae19-675eb13b7a78","Type":"ContainerStarted","Data":"bf9685458c936d447e8fabf47452ab43e020585b8fc969f105f0272e9e44f255"}
Oct 31 23:40:36.491831 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:36.491432    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5" event={"ID":"233555e0-e0b6-4610-ae19-675eb13b7a78","Type":"ContainerStarted","Data":"6d0ed3fc61262af40382ff7511dd00534b8f334599b0913d8f92e1b4b9ed7d01"}
Oct 31 23:40:36.491831 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:36.491515    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5"
Oct 31 23:40:36.511242 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:36.511200    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5" podStartSLOduration=1.995716284 podStartE2EDuration="2.511186458s" totalImagesPullingTime="515.470174ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:40:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:40:36.509516687 +0000 UTC m=+3055.256706204" watchObservedRunningTime="2026-10-31 23:40:36.511186458 +0000 UTC m=+3055.258375963"
Oct 31 23:40:36.835513 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:36.835431    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:40:42.500836 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:42.500808    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5"
Oct 31 23:40:53.711819 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:53.711782    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn"]
Oct 31 23:40:53.715495 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:53.715475    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn"
Oct 31 23:40:53.720591 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:53.720564    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-abd9b5\"/\"kube-root-ca.crt\""
Oct 31 23:40:53.720745 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:53.720728    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-abd9b5\"/\"simulator-single-main-abd9b5-kserve-self-signed-certs\""
Oct 31 23:40:53.720813 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:53.720730    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-abd9b5\"/\"openshift-service-ca.crt\""
Oct 31 23:40:53.720813 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:53.720756    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-abd9b5\"/\"default-dockercfg-tnpbp\""
Oct 31 23:40:53.733997 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:53.733935    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn"]
Oct 31 23:40:53.832409 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:53.832352    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a944f35-bb72-4aa5-8f51-0229a9919f90-home\") pod \"simulator-single-main-abd9b5-kserve-66bf785575-lbdxn\" (UID: \"1a944f35-bb72-4aa5-8f51-0229a9919f90\") " pod="e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn"
Oct 31 23:40:53.832614 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:53.832443    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a944f35-bb72-4aa5-8f51-0229a9919f90-tmp-dir\") pod \"simulator-single-main-abd9b5-kserve-66bf785575-lbdxn\" (UID: \"1a944f35-bb72-4aa5-8f51-0229a9919f90\") " pod="e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn"
Oct 31 23:40:53.832614 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:53.832550    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a944f35-bb72-4aa5-8f51-0229a9919f90-dshm\") pod \"simulator-single-main-abd9b5-kserve-66bf785575-lbdxn\" (UID: \"1a944f35-bb72-4aa5-8f51-0229a9919f90\") " pod="e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn"
Oct 31 23:40:53.832708 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:53.832626    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9xtd\" (UniqueName: \"kubernetes.io/projected/1a944f35-bb72-4aa5-8f51-0229a9919f90-kube-api-access-d9xtd\") pod \"simulator-single-main-abd9b5-kserve-66bf785575-lbdxn\" (UID: \"1a944f35-bb72-4aa5-8f51-0229a9919f90\") " pod="e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn"
Oct 31 23:40:53.832708 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:53.832700    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a944f35-bb72-4aa5-8f51-0229a9919f90-tls-certs\") pod \"simulator-single-main-abd9b5-kserve-66bf785575-lbdxn\" (UID: \"1a944f35-bb72-4aa5-8f51-0229a9919f90\") " pod="e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn"
Oct 31 23:40:53.832785 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:53.832768    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a944f35-bb72-4aa5-8f51-0229a9919f90-model-cache\") pod \"simulator-single-main-abd9b5-kserve-66bf785575-lbdxn\" (UID: \"1a944f35-bb72-4aa5-8f51-0229a9919f90\") " pod="e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn"
Oct 31 23:40:53.933424 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:53.933389    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a944f35-bb72-4aa5-8f51-0229a9919f90-home\") pod \"simulator-single-main-abd9b5-kserve-66bf785575-lbdxn\" (UID: \"1a944f35-bb72-4aa5-8f51-0229a9919f90\") " pod="e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn"
Oct 31 23:40:53.933639 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:53.933433    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a944f35-bb72-4aa5-8f51-0229a9919f90-tmp-dir\") pod \"simulator-single-main-abd9b5-kserve-66bf785575-lbdxn\" (UID: \"1a944f35-bb72-4aa5-8f51-0229a9919f90\") " pod="e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn"
Oct 31 23:40:53.933639 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:53.933450    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a944f35-bb72-4aa5-8f51-0229a9919f90-dshm\") pod \"simulator-single-main-abd9b5-kserve-66bf785575-lbdxn\" (UID: \"1a944f35-bb72-4aa5-8f51-0229a9919f90\") " pod="e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn"
Oct 31 23:40:53.933639 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:53.933469    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9xtd\" (UniqueName: \"kubernetes.io/projected/1a944f35-bb72-4aa5-8f51-0229a9919f90-kube-api-access-d9xtd\") pod \"simulator-single-main-abd9b5-kserve-66bf785575-lbdxn\" (UID: \"1a944f35-bb72-4aa5-8f51-0229a9919f90\") " pod="e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn"
Oct 31 23:40:53.933639 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:53.933521    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a944f35-bb72-4aa5-8f51-0229a9919f90-tls-certs\") pod \"simulator-single-main-abd9b5-kserve-66bf785575-lbdxn\" (UID: \"1a944f35-bb72-4aa5-8f51-0229a9919f90\") " pod="e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn"
Oct 31 23:40:53.933639 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:53.933557    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a944f35-bb72-4aa5-8f51-0229a9919f90-model-cache\") pod \"simulator-single-main-abd9b5-kserve-66bf785575-lbdxn\" (UID: \"1a944f35-bb72-4aa5-8f51-0229a9919f90\") " pod="e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn"
Oct 31 23:40:53.933937 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:53.933892    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a944f35-bb72-4aa5-8f51-0229a9919f90-home\") pod \"simulator-single-main-abd9b5-kserve-66bf785575-lbdxn\" (UID: \"1a944f35-bb72-4aa5-8f51-0229a9919f90\") " pod="e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn"
Oct 31 23:40:53.933937 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:53.933925    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a944f35-bb72-4aa5-8f51-0229a9919f90-tmp-dir\") pod \"simulator-single-main-abd9b5-kserve-66bf785575-lbdxn\" (UID: \"1a944f35-bb72-4aa5-8f51-0229a9919f90\") " pod="e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn"
Oct 31 23:40:53.934012 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:53.933981    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a944f35-bb72-4aa5-8f51-0229a9919f90-model-cache\") pod \"simulator-single-main-abd9b5-kserve-66bf785575-lbdxn\" (UID: \"1a944f35-bb72-4aa5-8f51-0229a9919f90\") " pod="e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn"
Oct 31 23:40:53.935851 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:53.935827    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a944f35-bb72-4aa5-8f51-0229a9919f90-dshm\") pod \"simulator-single-main-abd9b5-kserve-66bf785575-lbdxn\" (UID: \"1a944f35-bb72-4aa5-8f51-0229a9919f90\") " pod="e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn"
Oct 31 23:40:53.936036 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:53.935980    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a944f35-bb72-4aa5-8f51-0229a9919f90-tls-certs\") pod \"simulator-single-main-abd9b5-kserve-66bf785575-lbdxn\" (UID: \"1a944f35-bb72-4aa5-8f51-0229a9919f90\") " pod="e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn"
Oct 31 23:40:53.945280 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:53.945255    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9xtd\" (UniqueName: \"kubernetes.io/projected/1a944f35-bb72-4aa5-8f51-0229a9919f90-kube-api-access-d9xtd\") pod \"simulator-single-main-abd9b5-kserve-66bf785575-lbdxn\" (UID: \"1a944f35-bb72-4aa5-8f51-0229a9919f90\") " pod="e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn"
Oct 31 23:40:54.028233 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:54.028149    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn"
Oct 31 23:40:54.303484 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:54.303393    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn"]
Oct 31 23:40:54.307507 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:40:54.307476    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a944f35_bb72_4aa5_8f51_0229a9919f90.slice/crio-b5cad6450369bf229734c7958ef5537a50e79d880f58b70ca00ec78f1f4d92c7 WatchSource:0}: Error finding container b5cad6450369bf229734c7958ef5537a50e79d880f58b70ca00ec78f1f4d92c7: Status 404 returned error can't find the container with id b5cad6450369bf229734c7958ef5537a50e79d880f58b70ca00ec78f1f4d92c7
Oct 31 23:40:54.568834 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:54.568749    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn" event={"ID":"1a944f35-bb72-4aa5-8f51-0229a9919f90","Type":"ContainerStarted","Data":"f2fb9842b8f67a885142aa408067ab3eaaaef7e493cb309b6b0487ed9fe3bb12"}
Oct 31 23:40:54.568834 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:54.568800    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn" event={"ID":"1a944f35-bb72-4aa5-8f51-0229a9919f90","Type":"ContainerStarted","Data":"b5cad6450369bf229734c7958ef5537a50e79d880f58b70ca00ec78f1f4d92c7"}
Oct 31 23:40:54.585927 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:54.585873    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn" podStartSLOduration=1.5858593760000002 podStartE2EDuration="1.585859376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:40:54.585335026 +0000 UTC m=+3073.332524532" watchObservedRunningTime="2026-10-31 23:40:54.585859376 +0000 UTC m=+3073.333048880"
Oct 31 23:40:55.029235 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:55.029194    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn"
Oct 31 23:40:55.411403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:40:55.411368    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:41:02.545217 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:41:02.545171    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:41:04.028823 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:04.028767    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn"
Oct 31 23:41:04.041192 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:04.041168    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn"
Oct 31 23:41:04.624470 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:04.624441    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn"
Oct 31 23:41:08.419557 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:08.419521    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:41:32.188777 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:32.188697    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:41:32.545507 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:41:32.545414    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:41:36.924775 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:36.924740    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn"]
Oct 31 23:41:42.225234 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:42.225195    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5"]
Oct 31 23:41:42.225874 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:42.225494    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5" podUID="233555e0-e0b6-4610-ae19-675eb13b7a78" containerName="maas-api" containerID="cri-o://bf9685458c936d447e8fabf47452ab43e020585b8fc969f105f0272e9e44f255" gracePeriod=30
Oct 31 23:41:42.495978 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:42.495896    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5" podUID="233555e0-e0b6-4610-ae19-675eb13b7a78" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.140:8443/health\": dial tcp 10.133.0.140:8443: connect: connection refused"
Oct 31 23:41:43.461317 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:43.461292    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5"
Oct 31 23:41:43.558819 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:43.558788    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/233555e0-e0b6-4610-ae19-675eb13b7a78-metrics-certs\" (UniqueName: \"kubernetes.io/secret/233555e0-e0b6-4610-ae19-675eb13b7a78-metrics-certs\") pod \"233555e0-e0b6-4610-ae19-675eb13b7a78\" (UID: \"233555e0-e0b6-4610-ae19-675eb13b7a78\") "
Oct 31 23:41:43.559030 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:43.558900    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/233555e0-e0b6-4610-ae19-675eb13b7a78-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/233555e0-e0b6-4610-ae19-675eb13b7a78-openshift-service-ca\") pod \"233555e0-e0b6-4610-ae19-675eb13b7a78\" (UID: \"233555e0-e0b6-4610-ae19-675eb13b7a78\") "
Oct 31 23:41:43.559030 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:43.558929    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/233555e0-e0b6-4610-ae19-675eb13b7a78-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/233555e0-e0b6-4610-ae19-675eb13b7a78-maas-api-tls\") pod \"233555e0-e0b6-4610-ae19-675eb13b7a78\" (UID: \"233555e0-e0b6-4610-ae19-675eb13b7a78\") "
Oct 31 23:41:43.559030 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:43.559016    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/233555e0-e0b6-4610-ae19-675eb13b7a78-kube-api-access-dxtq9\" (UniqueName: \"kubernetes.io/projected/233555e0-e0b6-4610-ae19-675eb13b7a78-kube-api-access-dxtq9\") pod \"233555e0-e0b6-4610-ae19-675eb13b7a78\" (UID: \"233555e0-e0b6-4610-ae19-675eb13b7a78\") "
Oct 31 23:41:43.559315 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:43.559282    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/233555e0-e0b6-4610-ae19-675eb13b7a78-openshift-service-ca" pod "233555e0-e0b6-4610-ae19-675eb13b7a78" (UID: "233555e0-e0b6-4610-ae19-675eb13b7a78"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:41:43.560911 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:43.560882    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/233555e0-e0b6-4610-ae19-675eb13b7a78-metrics-certs" pod "233555e0-e0b6-4610-ae19-675eb13b7a78" (UID: "233555e0-e0b6-4610-ae19-675eb13b7a78"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:41:43.561071 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:43.561039    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/233555e0-e0b6-4610-ae19-675eb13b7a78-kube-api-access-dxtq9" pod "233555e0-e0b6-4610-ae19-675eb13b7a78" (UID: "233555e0-e0b6-4610-ae19-675eb13b7a78"). InnerVolumeSpecName "kube-api-access-dxtq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:41:43.561179 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:43.561090    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/233555e0-e0b6-4610-ae19-675eb13b7a78-maas-api-tls" pod "233555e0-e0b6-4610-ae19-675eb13b7a78" (UID: "233555e0-e0b6-4610-ae19-675eb13b7a78"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:41:43.660563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:43.660529    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/233555e0-e0b6-4610-ae19-675eb13b7a78-maas-api-tls\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:41:43.660563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:43.660558    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxtq9\" (UniqueName: \"kubernetes.io/projected/233555e0-e0b6-4610-ae19-675eb13b7a78-kube-api-access-dxtq9\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:41:43.660563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:43.660570    4301 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/233555e0-e0b6-4610-ae19-675eb13b7a78-metrics-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:41:43.660823 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:43.660578    4301 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/233555e0-e0b6-4610-ae19-675eb13b7a78-openshift-service-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:41:43.773395 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:43.772842    4301 generic.go:356] "Generic (PLEG): container finished" podID="233555e0-e0b6-4610-ae19-675eb13b7a78" containerID="bf9685458c936d447e8fabf47452ab43e020585b8fc969f105f0272e9e44f255" exitCode=0
Oct 31 23:41:43.773395 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:43.773027    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5" event={"ID":"233555e0-e0b6-4610-ae19-675eb13b7a78","Type":"ContainerDied","Data":"bf9685458c936d447e8fabf47452ab43e020585b8fc969f105f0272e9e44f255"}
Oct 31 23:41:43.773395 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:43.773109    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5" event={"ID":"233555e0-e0b6-4610-ae19-675eb13b7a78","Type":"ContainerDied","Data":"6d0ed3fc61262af40382ff7511dd00534b8f334599b0913d8f92e1b4b9ed7d01"}
Oct 31 23:41:43.773395 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:43.773152    4301 scope.go:122] "RemoveContainer" containerID="bf9685458c936d447e8fabf47452ab43e020585b8fc969f105f0272e9e44f255"
Oct 31 23:41:43.773395 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:43.773347    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5"
Oct 31 23:41:43.788693 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:43.788672    4301 scope.go:122] "RemoveContainer" containerID="bf9685458c936d447e8fabf47452ab43e020585b8fc969f105f0272e9e44f255"
Oct 31 23:41:43.788953 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:41:43.788937    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf9685458c936d447e8fabf47452ab43e020585b8fc969f105f0272e9e44f255\": container with ID starting with bf9685458c936d447e8fabf47452ab43e020585b8fc969f105f0272e9e44f255 not found: ID does not exist" containerID="bf9685458c936d447e8fabf47452ab43e020585b8fc969f105f0272e9e44f255"
Oct 31 23:41:43.789002 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:43.788961    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf9685458c936d447e8fabf47452ab43e020585b8fc969f105f0272e9e44f255"} err="failed to get container status \"bf9685458c936d447e8fabf47452ab43e020585b8fc969f105f0272e9e44f255\": rpc error: code = NotFound desc = could not find container \"bf9685458c936d447e8fabf47452ab43e020585b8fc969f105f0272e9e44f255\": container with ID starting with bf9685458c936d447e8fabf47452ab43e020585b8fc969f105f0272e9e44f255 not found: ID does not exist"
Oct 31 23:41:43.796599 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:43.796578    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5"]
Oct 31 23:41:43.801617 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:43.801591    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-abd9b5-8588d6c659-mjlg5"]
Oct 31 23:41:43.912722 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:43.912645    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:41:45.401601 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:45.401571    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="233555e0-e0b6-4610-ae19-675eb13b7a78" path="/var/lib/kubelet/pods/233555e0-e0b6-4610-ae19-675eb13b7a78/volumes"
Oct 31 23:41:51.944747 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:51.944702    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn" podUID="1a944f35-bb72-4aa5-8f51-0229a9919f90" containerName="main" containerID="cri-o://f2fb9842b8f67a885142aa408067ab3eaaaef7e493cb309b6b0487ed9fe3bb12" gracePeriod=45
Oct 31 23:41:52.186414 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:52.186391    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn"
Oct 31 23:41:52.336887 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:52.336796    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a944f35-bb72-4aa5-8f51-0229a9919f90-kube-api-access-d9xtd\" (UniqueName: \"kubernetes.io/projected/1a944f35-bb72-4aa5-8f51-0229a9919f90-kube-api-access-d9xtd\") pod \"1a944f35-bb72-4aa5-8f51-0229a9919f90\" (UID: \"1a944f35-bb72-4aa5-8f51-0229a9919f90\") "
Oct 31 23:41:52.336887 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:52.336865    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a944f35-bb72-4aa5-8f51-0229a9919f90-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a944f35-bb72-4aa5-8f51-0229a9919f90-model-cache\") pod \"1a944f35-bb72-4aa5-8f51-0229a9919f90\" (UID: \"1a944f35-bb72-4aa5-8f51-0229a9919f90\") "
Oct 31 23:41:52.337122 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:52.336896    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a944f35-bb72-4aa5-8f51-0229a9919f90-home\" (UniqueName: \"kubernetes.io/empty-dir/1a944f35-bb72-4aa5-8f51-0229a9919f90-home\") pod \"1a944f35-bb72-4aa5-8f51-0229a9919f90\" (UID: \"1a944f35-bb72-4aa5-8f51-0229a9919f90\") "
Oct 31 23:41:52.337122 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:52.336918    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a944f35-bb72-4aa5-8f51-0229a9919f90-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a944f35-bb72-4aa5-8f51-0229a9919f90-dshm\") pod \"1a944f35-bb72-4aa5-8f51-0229a9919f90\" (UID: \"1a944f35-bb72-4aa5-8f51-0229a9919f90\") "
Oct 31 23:41:52.337122 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:52.336947    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a944f35-bb72-4aa5-8f51-0229a9919f90-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a944f35-bb72-4aa5-8f51-0229a9919f90-tmp-dir\") pod \"1a944f35-bb72-4aa5-8f51-0229a9919f90\" (UID: \"1a944f35-bb72-4aa5-8f51-0229a9919f90\") "
Oct 31 23:41:52.337122 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:52.336965    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a944f35-bb72-4aa5-8f51-0229a9919f90-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a944f35-bb72-4aa5-8f51-0229a9919f90-tls-certs\") pod \"1a944f35-bb72-4aa5-8f51-0229a9919f90\" (UID: \"1a944f35-bb72-4aa5-8f51-0229a9919f90\") "
Oct 31 23:41:52.337340 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:52.337139    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a944f35-bb72-4aa5-8f51-0229a9919f90-model-cache" pod "1a944f35-bb72-4aa5-8f51-0229a9919f90" (UID: "1a944f35-bb72-4aa5-8f51-0229a9919f90"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:41:52.337340 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:52.337232    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a944f35-bb72-4aa5-8f51-0229a9919f90-home" pod "1a944f35-bb72-4aa5-8f51-0229a9919f90" (UID: "1a944f35-bb72-4aa5-8f51-0229a9919f90"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:41:52.337340 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:52.337274    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a944f35-bb72-4aa5-8f51-0229a9919f90-tmp-dir" pod "1a944f35-bb72-4aa5-8f51-0229a9919f90" (UID: "1a944f35-bb72-4aa5-8f51-0229a9919f90"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:41:52.339168 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:52.339138    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a944f35-bb72-4aa5-8f51-0229a9919f90-kube-api-access-d9xtd" pod "1a944f35-bb72-4aa5-8f51-0229a9919f90" (UID: "1a944f35-bb72-4aa5-8f51-0229a9919f90"). InnerVolumeSpecName "kube-api-access-d9xtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:41:52.339168 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:52.339152    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a944f35-bb72-4aa5-8f51-0229a9919f90-tls-certs" pod "1a944f35-bb72-4aa5-8f51-0229a9919f90" (UID: "1a944f35-bb72-4aa5-8f51-0229a9919f90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:41:52.339322 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:52.339212    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a944f35-bb72-4aa5-8f51-0229a9919f90-dshm" pod "1a944f35-bb72-4aa5-8f51-0229a9919f90" (UID: "1a944f35-bb72-4aa5-8f51-0229a9919f90"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:41:52.437583 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:52.437554    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9xtd\" (UniqueName: \"kubernetes.io/projected/1a944f35-bb72-4aa5-8f51-0229a9919f90-kube-api-access-d9xtd\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:41:52.437583 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:52.437582    4301 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a944f35-bb72-4aa5-8f51-0229a9919f90-model-cache\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:41:52.437784 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:52.437593    4301 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a944f35-bb72-4aa5-8f51-0229a9919f90-home\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:41:52.437784 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:52.437603    4301 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a944f35-bb72-4aa5-8f51-0229a9919f90-dshm\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:41:52.437784 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:52.437643    4301 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a944f35-bb72-4aa5-8f51-0229a9919f90-tmp-dir\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:41:52.437784 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:52.437652    4301 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a944f35-bb72-4aa5-8f51-0229a9919f90-tls-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:41:52.810791 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:52.810754    4301 generic.go:356] "Generic (PLEG): container finished" podID="1a944f35-bb72-4aa5-8f51-0229a9919f90" containerID="f2fb9842b8f67a885142aa408067ab3eaaaef7e493cb309b6b0487ed9fe3bb12" exitCode=0
Oct 31 23:41:52.810976 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:52.810801    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn"
Oct 31 23:41:52.810976 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:52.810818    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn" event={"ID":"1a944f35-bb72-4aa5-8f51-0229a9919f90","Type":"ContainerDied","Data":"f2fb9842b8f67a885142aa408067ab3eaaaef7e493cb309b6b0487ed9fe3bb12"}
Oct 31 23:41:52.810976 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:52.810857    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn" event={"ID":"1a944f35-bb72-4aa5-8f51-0229a9919f90","Type":"ContainerDied","Data":"b5cad6450369bf229734c7958ef5537a50e79d880f58b70ca00ec78f1f4d92c7"}
Oct 31 23:41:52.810976 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:52.810879    4301 scope.go:122] "RemoveContainer" containerID="f2fb9842b8f67a885142aa408067ab3eaaaef7e493cb309b6b0487ed9fe3bb12"
Oct 31 23:41:52.819916 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:52.819895    4301 scope.go:122] "RemoveContainer" containerID="f2fb9842b8f67a885142aa408067ab3eaaaef7e493cb309b6b0487ed9fe3bb12"
Oct 31 23:41:52.820198 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:41:52.820180    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2fb9842b8f67a885142aa408067ab3eaaaef7e493cb309b6b0487ed9fe3bb12\": container with ID starting with f2fb9842b8f67a885142aa408067ab3eaaaef7e493cb309b6b0487ed9fe3bb12 not found: ID does not exist" containerID="f2fb9842b8f67a885142aa408067ab3eaaaef7e493cb309b6b0487ed9fe3bb12"
Oct 31 23:41:52.820237 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:52.820210    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2fb9842b8f67a885142aa408067ab3eaaaef7e493cb309b6b0487ed9fe3bb12"} err="failed to get container status \"f2fb9842b8f67a885142aa408067ab3eaaaef7e493cb309b6b0487ed9fe3bb12\": rpc error: code = NotFound desc = could not find container \"f2fb9842b8f67a885142aa408067ab3eaaaef7e493cb309b6b0487ed9fe3bb12\": container with ID starting with f2fb9842b8f67a885142aa408067ab3eaaaef7e493cb309b6b0487ed9fe3bb12 not found: ID does not exist"
Oct 31 23:41:52.831205 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:52.831176    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn"]
Oct 31 23:41:52.837271 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:52.837247    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-abd9b5/simulator-single-main-abd9b5-kserve-66bf785575-lbdxn"]
Oct 31 23:41:53.402067 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:53.402032    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a944f35-bb72-4aa5-8f51-0229a9919f90" path="/var/lib/kubelet/pods/1a944f35-bb72-4aa5-8f51-0229a9919f90/volumes"
Oct 31 23:41:53.904280 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:53.904246    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l"]
Oct 31 23:41:53.904620 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:53.904607    4301 cpu_manager.go:414] "containerMap: removing container" podUID="233555e0-e0b6-4610-ae19-675eb13b7a78" containerName="maas-api"
Oct 31 23:41:53.904680 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:53.904622    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="233555e0-e0b6-4610-ae19-675eb13b7a78" containerName="maas-api"
Oct 31 23:41:53.904680 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:53.904632    4301 cpu_manager.go:414] "containerMap: removing container" podUID="1a944f35-bb72-4aa5-8f51-0229a9919f90" containerName="main"
Oct 31 23:41:53.904680 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:53.904637    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a944f35-bb72-4aa5-8f51-0229a9919f90" containerName="main"
Oct 31 23:41:53.904792 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:53.904689    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="233555e0-e0b6-4610-ae19-675eb13b7a78" containerName="maas-api"
Oct 31 23:41:53.904792 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:53.904697    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a944f35-bb72-4aa5-8f51-0229a9919f90" containerName="main"
Oct 31 23:41:53.909087 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:53.909064    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l"
Oct 31 23:41:53.917942 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:53.917918    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 31 23:41:53.918291 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:53.918269    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 31 23:41:53.918540 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:53.918523    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ltzbh\""
Oct 31 23:41:53.918622 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:53.918606    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-49bfba53-kserve-self-signed-certs\""
Oct 31 23:41:53.920821 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:53.920800    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l"]
Oct 31 23:41:54.051937 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:54.051897    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-model-cache\") pod \"e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l\" (UID: \"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590\") " pod="llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l"
Oct 31 23:41:54.052146 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:54.052020    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-dshm\") pod \"e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l\" (UID: \"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590\") " pod="llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l"
Oct 31 23:41:54.052146 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:54.052102    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-tmp-dir\") pod \"e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l\" (UID: \"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590\") " pod="llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l"
Oct 31 23:41:54.052271 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:54.052177    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-home\") pod \"e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l\" (UID: \"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590\") " pod="llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l"
Oct 31 23:41:54.052271 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:54.052219    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-tls-certs\") pod \"e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l\" (UID: \"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590\") " pod="llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l"
Oct 31 23:41:54.052271 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:54.052255    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mjzn\" (UniqueName: \"kubernetes.io/projected/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-kube-api-access-4mjzn\") pod \"e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l\" (UID: \"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590\") " pod="llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l"
Oct 31 23:41:54.153872 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:54.153833    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-model-cache\") pod \"e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l\" (UID: \"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590\") " pod="llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l"
Oct 31 23:41:54.154069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:54.153881    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-dshm\") pod \"e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l\" (UID: \"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590\") " pod="llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l"
Oct 31 23:41:54.154069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:54.153903    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-tmp-dir\") pod \"e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l\" (UID: \"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590\") " pod="llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l"
Oct 31 23:41:54.154069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:54.153937    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-home\") pod \"e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l\" (UID: \"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590\") " pod="llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l"
Oct 31 23:41:54.154069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:54.153992    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-tls-certs\") pod \"e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l\" (UID: \"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590\") " pod="llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l"
Oct 31 23:41:54.154069 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:54.154023    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mjzn\" (UniqueName: \"kubernetes.io/projected/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-kube-api-access-4mjzn\") pod \"e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l\" (UID: \"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590\") " pod="llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l"
Oct 31 23:41:54.154395 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:54.154318    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-model-cache\") pod \"e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l\" (UID: \"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590\") " pod="llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l"
Oct 31 23:41:54.154395 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:54.154389    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-home\") pod \"e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l\" (UID: \"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590\") " pod="llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l"
Oct 31 23:41:54.154514 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:54.154453    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-tmp-dir\") pod \"e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l\" (UID: \"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590\") " pod="llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l"
Oct 31 23:41:54.156340 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:54.156317    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-dshm\") pod \"e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l\" (UID: \"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590\") " pod="llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l"
Oct 31 23:41:54.156596 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:54.156580    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-tls-certs\") pod \"e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l\" (UID: \"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590\") " pod="llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l"
Oct 31 23:41:54.165770 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:54.165745    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mjzn\" (UniqueName: \"kubernetes.io/projected/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-kube-api-access-4mjzn\") pod \"e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l\" (UID: \"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590\") " pod="llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l"
Oct 31 23:41:54.221497 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:54.221467    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l"
Oct 31 23:41:54.499780 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:54.499693    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l"]
Oct 31 23:41:54.503529 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:41:54.503497    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5e3bcf5_4f41_4f93_9aa0_edfcab0da590.slice/crio-f5a0af8d9e962df22539aee0def984d31ef3cf45ede237490904d0d1fdb52a31 WatchSource:0}: Error finding container f5a0af8d9e962df22539aee0def984d31ef3cf45ede237490904d0d1fdb52a31: Status 404 returned error can't find the container with id f5a0af8d9e962df22539aee0def984d31ef3cf45ede237490904d0d1fdb52a31
Oct 31 23:41:54.822774 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:54.822682    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l" event={"ID":"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590","Type":"ContainerStarted","Data":"5556d0acfb7160d59f9addcdb8d0babfd712a9eff3dd6aebc17388fbba071356"}
Oct 31 23:41:54.822774 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:54.822725    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l" event={"ID":"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590","Type":"ContainerStarted","Data":"f5a0af8d9e962df22539aee0def984d31ef3cf45ede237490904d0d1fdb52a31"}
Oct 31 23:41:54.841885 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:54.841830    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l" podStartSLOduration=1.8418176389999998 podStartE2EDuration="1.841817639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:41:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:41:54.839882365 +0000 UTC m=+3133.587071869" watchObservedRunningTime="2026-10-31 23:41:54.841817639 +0000 UTC m=+3133.589007191"
Oct 31 23:41:55.222246 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:55.222209    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l"
Oct 31 23:41:55.387717 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:41:55.387687    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:42:02.544895 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:42:02.544853    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:42:04.222672 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:04.222633    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l"
Oct 31 23:42:04.234994 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:04.234960    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l"
Oct 31 23:42:04.889564 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:04.889525    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l"
Oct 31 23:42:10.403954 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:10.403911    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn"]
Oct 31 23:42:10.408270 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:10.408248    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn"
Oct 31 23:42:10.416346 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:10.416324    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-49bfba53-kserve-self-signed-certs\""
Oct 31 23:42:10.418633 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:10.418605    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn"]
Oct 31 23:42:10.422872 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:10.422846    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrlv6\" (UniqueName: \"kubernetes.io/projected/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-kube-api-access-mrlv6\") pod \"e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn\" (UID: \"ccd8d0e3-97e2-4610-ba6a-1a8af623b020\") " pod="llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn"
Oct 31 23:42:10.423002 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:10.422885    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-tls-certs\") pod \"e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn\" (UID: \"ccd8d0e3-97e2-4610-ba6a-1a8af623b020\") " pod="llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn"
Oct 31 23:42:10.423002 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:10.422910    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-tmp-dir\") pod \"e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn\" (UID: \"ccd8d0e3-97e2-4610-ba6a-1a8af623b020\") " pod="llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn"
Oct 31 23:42:10.423002 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:10.422941    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-model-cache\") pod \"e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn\" (UID: \"ccd8d0e3-97e2-4610-ba6a-1a8af623b020\") " pod="llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn"
Oct 31 23:42:10.423002 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:10.422992    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-home\") pod \"e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn\" (UID: \"ccd8d0e3-97e2-4610-ba6a-1a8af623b020\") " pod="llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn"
Oct 31 23:42:10.423215 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:10.423067    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-dshm\") pod \"e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn\" (UID: \"ccd8d0e3-97e2-4610-ba6a-1a8af623b020\") " pod="llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn"
Oct 31 23:42:10.523814 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:10.523780    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrlv6\" (UniqueName: \"kubernetes.io/projected/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-kube-api-access-mrlv6\") pod \"e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn\" (UID: \"ccd8d0e3-97e2-4610-ba6a-1a8af623b020\") " pod="llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn"
Oct 31 23:42:10.523814 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:10.523814    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-tls-certs\") pod \"e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn\" (UID: \"ccd8d0e3-97e2-4610-ba6a-1a8af623b020\") " pod="llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn"
Oct 31 23:42:10.524081 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:10.523833    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-tmp-dir\") pod \"e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn\" (UID: \"ccd8d0e3-97e2-4610-ba6a-1a8af623b020\") " pod="llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn"
Oct 31 23:42:10.524081 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:10.523861    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-model-cache\") pod \"e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn\" (UID: \"ccd8d0e3-97e2-4610-ba6a-1a8af623b020\") " pod="llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn"
Oct 31 23:42:10.524081 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:10.523911    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-home\") pod \"e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn\" (UID: \"ccd8d0e3-97e2-4610-ba6a-1a8af623b020\") " pod="llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn"
Oct 31 23:42:10.524081 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:10.523965    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-dshm\") pod \"e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn\" (UID: \"ccd8d0e3-97e2-4610-ba6a-1a8af623b020\") " pod="llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn"
Oct 31 23:42:10.524287 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:10.524237    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-tmp-dir\") pod \"e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn\" (UID: \"ccd8d0e3-97e2-4610-ba6a-1a8af623b020\") " pod="llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn"
Oct 31 23:42:10.524489 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:10.524465    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-home\") pod \"e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn\" (UID: \"ccd8d0e3-97e2-4610-ba6a-1a8af623b020\") " pod="llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn"
Oct 31 23:42:10.524562 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:10.524539    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-model-cache\") pod \"e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn\" (UID: \"ccd8d0e3-97e2-4610-ba6a-1a8af623b020\") " pod="llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn"
Oct 31 23:42:10.526813 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:10.526762    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-dshm\") pod \"e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn\" (UID: \"ccd8d0e3-97e2-4610-ba6a-1a8af623b020\") " pod="llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn"
Oct 31 23:42:10.527740 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:10.527713    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-tls-certs\") pod \"e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn\" (UID: \"ccd8d0e3-97e2-4610-ba6a-1a8af623b020\") " pod="llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn"
Oct 31 23:42:10.535756 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:10.535736    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrlv6\" (UniqueName: \"kubernetes.io/projected/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-kube-api-access-mrlv6\") pod \"e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn\" (UID: \"ccd8d0e3-97e2-4610-ba6a-1a8af623b020\") " pod="llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn"
Oct 31 23:42:10.722740 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:10.722649    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn"
Oct 31 23:42:10.999111 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:10.999039    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn"]
Oct 31 23:42:11.002581 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:42:11.002548    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccd8d0e3_97e2_4610_ba6a_1a8af623b020.slice/crio-16b821726f39a8e196a333d2b7177eba91258bd736975f8987e74437f2c36d5a WatchSource:0}: Error finding container 16b821726f39a8e196a333d2b7177eba91258bd736975f8987e74437f2c36d5a: Status 404 returned error can't find the container with id 16b821726f39a8e196a333d2b7177eba91258bd736975f8987e74437f2c36d5a
Oct 31 23:42:11.909496 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:11.909449    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn" event={"ID":"ccd8d0e3-97e2-4610-ba6a-1a8af623b020","Type":"ContainerStarted","Data":"f454e09f61c3197e551136c51666b3f978c6e6bd6429e7cb279496af7e9aa3bb"}
Oct 31 23:42:11.909954 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:11.909509    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn" event={"ID":"ccd8d0e3-97e2-4610-ba6a-1a8af623b020","Type":"ContainerStarted","Data":"16b821726f39a8e196a333d2b7177eba91258bd736975f8987e74437f2c36d5a"}
Oct 31 23:42:11.927235 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:11.927188    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn" podStartSLOduration=1.927175579 podStartE2EDuration="1.927175579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:42:11.926653558 +0000 UTC m=+3150.673843061" watchObservedRunningTime="2026-10-31 23:42:11.927175579 +0000 UTC m=+3150.674365085"
Oct 31 23:42:12.022577 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:12.022544    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:42:12.723769 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:12.723722    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn"
Oct 31 23:42:20.723205 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:20.723163    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn"
Oct 31 23:42:20.737895 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:20.737863    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn"
Oct 31 23:42:20.960858 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:20.960829    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn"
Oct 31 23:42:27.804298 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:27.804258    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:42:32.544851 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:42:32.544814    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:42:38.290305 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:38.290257    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l"]
Oct 31 23:42:39.619594 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:39.619552    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:42:40.378663 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:40.378629    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn"]
Oct 31 23:42:45.860386 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:45.860085    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:42:53.310234 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:53.310191    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l" podUID="a5e3bcf5-4f41-4f93-9aa0-edfcab0da590" containerName="main" containerID="cri-o://5556d0acfb7160d59f9addcdb8d0babfd712a9eff3dd6aebc17388fbba071356" gracePeriod=45
Oct 31 23:42:53.549811 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:53.549784    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l"
Oct 31 23:42:53.663476 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:53.663444    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-model-cache\") pod \"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590\" (UID: \"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590\") "
Oct 31 23:42:53.663682 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:53.663511    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-dshm\") pod \"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590\" (UID: \"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590\") "
Oct 31 23:42:53.663682 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:53.663563    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-tmp-dir\") pod \"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590\" (UID: \"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590\") "
Oct 31 23:42:53.663682 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:53.663587    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-home\" (UniqueName: \"kubernetes.io/empty-dir/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-home\") pod \"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590\" (UID: \"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590\") "
Oct 31 23:42:53.663682 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:53.663615    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-tls-certs\" (UniqueName: \"kubernetes.io/secret/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-tls-certs\") pod \"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590\" (UID: \"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590\") "
Oct 31 23:42:53.663682 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:53.663648    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-kube-api-access-4mjzn\" (UniqueName: \"kubernetes.io/projected/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-kube-api-access-4mjzn\") pod \"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590\" (UID: \"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590\") "
Oct 31 23:42:53.663952 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:53.663789    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-model-cache" pod "a5e3bcf5-4f41-4f93-9aa0-edfcab0da590" (UID: "a5e3bcf5-4f41-4f93-9aa0-edfcab0da590"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:42:53.663952 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:53.663884    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-tmp-dir" pod "a5e3bcf5-4f41-4f93-9aa0-edfcab0da590" (UID: "a5e3bcf5-4f41-4f93-9aa0-edfcab0da590"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:42:53.663952 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:53.663898    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-home" pod "a5e3bcf5-4f41-4f93-9aa0-edfcab0da590" (UID: "a5e3bcf5-4f41-4f93-9aa0-edfcab0da590"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:42:53.664111 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:53.664022    4301 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-model-cache\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:42:53.664111 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:53.664065    4301 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-tmp-dir\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:42:53.664111 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:53.664082    4301 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-home\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:42:53.665924 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:53.665894    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-dshm" pod "a5e3bcf5-4f41-4f93-9aa0-edfcab0da590" (UID: "a5e3bcf5-4f41-4f93-9aa0-edfcab0da590"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:42:53.666040 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:53.666000    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-kube-api-access-4mjzn" pod "a5e3bcf5-4f41-4f93-9aa0-edfcab0da590" (UID: "a5e3bcf5-4f41-4f93-9aa0-edfcab0da590"). InnerVolumeSpecName "kube-api-access-4mjzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:42:53.666040 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:53.666016    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-tls-certs" pod "a5e3bcf5-4f41-4f93-9aa0-edfcab0da590" (UID: "a5e3bcf5-4f41-4f93-9aa0-edfcab0da590"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:42:53.764961 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:53.764928    4301 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-dshm\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:42:53.764961 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:53.764956    4301 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-tls-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:42:53.764961 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:53.764966    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mjzn\" (UniqueName: \"kubernetes.io/projected/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590-kube-api-access-4mjzn\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:42:54.132012 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:54.131980    4301 generic.go:356] "Generic (PLEG): container finished" podID="a5e3bcf5-4f41-4f93-9aa0-edfcab0da590" containerID="5556d0acfb7160d59f9addcdb8d0babfd712a9eff3dd6aebc17388fbba071356" exitCode=0
Oct 31 23:42:54.132188 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:54.132029    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l"
Oct 31 23:42:54.132188 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:54.132069    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l" event={"ID":"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590","Type":"ContainerDied","Data":"5556d0acfb7160d59f9addcdb8d0babfd712a9eff3dd6aebc17388fbba071356"}
Oct 31 23:42:54.132188 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:54.132120    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l" event={"ID":"a5e3bcf5-4f41-4f93-9aa0-edfcab0da590","Type":"ContainerDied","Data":"f5a0af8d9e962df22539aee0def984d31ef3cf45ede237490904d0d1fdb52a31"}
Oct 31 23:42:54.132188 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:54.132141    4301 scope.go:122] "RemoveContainer" containerID="5556d0acfb7160d59f9addcdb8d0babfd712a9eff3dd6aebc17388fbba071356"
Oct 31 23:42:54.140695 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:54.140679    4301 scope.go:122] "RemoveContainer" containerID="5556d0acfb7160d59f9addcdb8d0babfd712a9eff3dd6aebc17388fbba071356"
Oct 31 23:42:54.140905 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:42:54.140886    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5556d0acfb7160d59f9addcdb8d0babfd712a9eff3dd6aebc17388fbba071356\": container with ID starting with 5556d0acfb7160d59f9addcdb8d0babfd712a9eff3dd6aebc17388fbba071356 not found: ID does not exist" containerID="5556d0acfb7160d59f9addcdb8d0babfd712a9eff3dd6aebc17388fbba071356"
Oct 31 23:42:54.140972 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:54.140913    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5556d0acfb7160d59f9addcdb8d0babfd712a9eff3dd6aebc17388fbba071356"} err="failed to get container status \"5556d0acfb7160d59f9addcdb8d0babfd712a9eff3dd6aebc17388fbba071356\": rpc error: code = NotFound desc = could not find container \"5556d0acfb7160d59f9addcdb8d0babfd712a9eff3dd6aebc17388fbba071356\": container with ID starting with 5556d0acfb7160d59f9addcdb8d0babfd712a9eff3dd6aebc17388fbba071356 not found: ID does not exist"
Oct 31 23:42:54.151582 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:54.151562    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l"]
Oct 31 23:42:54.155543 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:54.155524    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-49bfba53-kserve-6bb577cdd9-6hb8l"]
Oct 31 23:42:54.512618 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:54.512541    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:42:55.405401 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:55.405340    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5e3bcf5-4f41-4f93-9aa0-edfcab0da590" path="/var/lib/kubelet/pods/a5e3bcf5-4f41-4f93-9aa0-edfcab0da590/volumes"
Oct 31 23:42:55.410842 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:55.410216    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn" podUID="ccd8d0e3-97e2-4610-ba6a-1a8af623b020" containerName="main" containerID="cri-o://f454e09f61c3197e551136c51666b3f978c6e6bd6429e7cb279496af7e9aa3bb" gracePeriod=45
Oct 31 23:42:55.851381 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:55.850558    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn"
Oct 31 23:42:55.901546 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:55.900655    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-tmp-dir\") pod \"ccd8d0e3-97e2-4610-ba6a-1a8af623b020\" (UID: \"ccd8d0e3-97e2-4610-ba6a-1a8af623b020\") "
Oct 31 23:42:55.901546 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:55.900745    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-tls-certs\" (UniqueName: \"kubernetes.io/secret/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-tls-certs\") pod \"ccd8d0e3-97e2-4610-ba6a-1a8af623b020\" (UID: \"ccd8d0e3-97e2-4610-ba6a-1a8af623b020\") "
Oct 31 23:42:55.901546 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:55.900792    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-home\" (UniqueName: \"kubernetes.io/empty-dir/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-home\") pod \"ccd8d0e3-97e2-4610-ba6a-1a8af623b020\" (UID: \"ccd8d0e3-97e2-4610-ba6a-1a8af623b020\") "
Oct 31 23:42:55.901546 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:55.900825    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-dshm\") pod \"ccd8d0e3-97e2-4610-ba6a-1a8af623b020\" (UID: \"ccd8d0e3-97e2-4610-ba6a-1a8af623b020\") "
Oct 31 23:42:55.901546 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:55.900887    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-model-cache\") pod \"ccd8d0e3-97e2-4610-ba6a-1a8af623b020\" (UID: \"ccd8d0e3-97e2-4610-ba6a-1a8af623b020\") "
Oct 31 23:42:55.901546 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:55.900920    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-kube-api-access-mrlv6\" (UniqueName: \"kubernetes.io/projected/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-kube-api-access-mrlv6\") pod \"ccd8d0e3-97e2-4610-ba6a-1a8af623b020\" (UID: \"ccd8d0e3-97e2-4610-ba6a-1a8af623b020\") "
Oct 31 23:42:55.910380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:55.903933    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-kube-api-access-mrlv6" pod "ccd8d0e3-97e2-4610-ba6a-1a8af623b020" (UID: "ccd8d0e3-97e2-4610-ba6a-1a8af623b020"). InnerVolumeSpecName "kube-api-access-mrlv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:42:55.911494 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:55.911451    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-tmp-dir" pod "ccd8d0e3-97e2-4610-ba6a-1a8af623b020" (UID: "ccd8d0e3-97e2-4610-ba6a-1a8af623b020"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:42:55.911674 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:55.911643    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-model-cache" pod "ccd8d0e3-97e2-4610-ba6a-1a8af623b020" (UID: "ccd8d0e3-97e2-4610-ba6a-1a8af623b020"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:42:55.911827 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:55.911809    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-home" pod "ccd8d0e3-97e2-4610-ba6a-1a8af623b020" (UID: "ccd8d0e3-97e2-4610-ba6a-1a8af623b020"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:42:55.913484 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:55.913329    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-dshm" pod "ccd8d0e3-97e2-4610-ba6a-1a8af623b020" (UID: "ccd8d0e3-97e2-4610-ba6a-1a8af623b020"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:42:55.914849 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:55.914813    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-tls-certs" pod "ccd8d0e3-97e2-4610-ba6a-1a8af623b020" (UID: "ccd8d0e3-97e2-4610-ba6a-1a8af623b020"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:42:56.002471 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:56.002250    4301 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-tmp-dir\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:42:56.002471 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:56.002295    4301 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-tls-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:42:56.002471 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:56.002309    4301 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-home\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:42:56.002471 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:56.002323    4301 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-dshm\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:42:56.002471 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:56.002336    4301 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-model-cache\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:42:56.002471 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:56.002351    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrlv6\" (UniqueName: \"kubernetes.io/projected/ccd8d0e3-97e2-4610-ba6a-1a8af623b020-kube-api-access-mrlv6\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:42:56.172388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:56.171927    4301 generic.go:356] "Generic (PLEG): container finished" podID="ccd8d0e3-97e2-4610-ba6a-1a8af623b020" containerID="f454e09f61c3197e551136c51666b3f978c6e6bd6429e7cb279496af7e9aa3bb" exitCode=0
Oct 31 23:42:56.172388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:56.172054    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn" event={"ID":"ccd8d0e3-97e2-4610-ba6a-1a8af623b020","Type":"ContainerDied","Data":"f454e09f61c3197e551136c51666b3f978c6e6bd6429e7cb279496af7e9aa3bb"}
Oct 31 23:42:56.172388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:56.172092    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn" event={"ID":"ccd8d0e3-97e2-4610-ba6a-1a8af623b020","Type":"ContainerDied","Data":"16b821726f39a8e196a333d2b7177eba91258bd736975f8987e74437f2c36d5a"}
Oct 31 23:42:56.172388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:56.172114    4301 scope.go:122] "RemoveContainer" containerID="f454e09f61c3197e551136c51666b3f978c6e6bd6429e7cb279496af7e9aa3bb"
Oct 31 23:42:56.172388 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:56.172223    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn"
Oct 31 23:42:56.213061 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:56.213035    4301 scope.go:122] "RemoveContainer" containerID="f454e09f61c3197e551136c51666b3f978c6e6bd6429e7cb279496af7e9aa3bb"
Oct 31 23:42:56.213415 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:42:56.213380    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f454e09f61c3197e551136c51666b3f978c6e6bd6429e7cb279496af7e9aa3bb\": container with ID starting with f454e09f61c3197e551136c51666b3f978c6e6bd6429e7cb279496af7e9aa3bb not found: ID does not exist" containerID="f454e09f61c3197e551136c51666b3f978c6e6bd6429e7cb279496af7e9aa3bb"
Oct 31 23:42:56.213504 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:56.213424    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f454e09f61c3197e551136c51666b3f978c6e6bd6429e7cb279496af7e9aa3bb"} err="failed to get container status \"f454e09f61c3197e551136c51666b3f978c6e6bd6429e7cb279496af7e9aa3bb\": rpc error: code = NotFound desc = could not find container \"f454e09f61c3197e551136c51666b3f978c6e6bd6429e7cb279496af7e9aa3bb\": container with ID starting with f454e09f61c3197e551136c51666b3f978c6e6bd6429e7cb279496af7e9aa3bb not found: ID does not exist"
Oct 31 23:42:56.219912 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:56.219890    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn"]
Oct 31 23:42:56.223592 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:56.223572    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-49bfba53-kserve-8c5778f7-6d6gn"]
Oct 31 23:42:57.403169 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:57.403125    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccd8d0e3-97e2-4610-ba6a-1a8af623b020" path="/var/lib/kubelet/pods/ccd8d0e3-97e2-4610-ba6a-1a8af623b020/volumes"
Oct 31 23:42:59.370539 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:59.370497    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:42:59.550560 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:59.550504    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-s6nvl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 23:42:59.550929 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:42:59.550895    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl" podUID="99873d5d-87c2-423a-9a44-1b3fc83d0b46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:43:02.544942 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:43:02.544902    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:43:15.022041 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:15.022007    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:43:16.516921 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:16.516884    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6bf6d56f9f-9zkqk"]
Oct 31 23:43:16.517483 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:16.517229    4301 cpu_manager.go:414] "containerMap: removing container" podUID="a5e3bcf5-4f41-4f93-9aa0-edfcab0da590" containerName="main"
Oct 31 23:43:16.517483 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:16.517240    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5e3bcf5-4f41-4f93-9aa0-edfcab0da590" containerName="main"
Oct 31 23:43:16.517483 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:16.517252    4301 cpu_manager.go:414] "containerMap: removing container" podUID="ccd8d0e3-97e2-4610-ba6a-1a8af623b020" containerName="main"
Oct 31 23:43:16.517483 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:16.517257    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd8d0e3-97e2-4610-ba6a-1a8af623b020" containerName="main"
Oct 31 23:43:16.517483 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:16.517306    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5e3bcf5-4f41-4f93-9aa0-edfcab0da590" containerName="main"
Oct 31 23:43:16.517483 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:16.517313    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccd8d0e3-97e2-4610-ba6a-1a8af623b020" containerName="main"
Oct 31 23:43:16.520300 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:16.520278    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-9zkqk"
Oct 31 23:43:16.532260 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:16.532194    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6bf6d56f9f-9zkqk"]
Oct 31 23:43:16.632908 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:16.632872    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24ca21f3-521c-4bf8-80b4-1c0a7757a0bd-maas-api-tls\") pod \"maas-api-6bf6d56f9f-9zkqk\" (UID: \"24ca21f3-521c-4bf8-80b4-1c0a7757a0bd\") " pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-9zkqk"
Oct 31 23:43:16.633110 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:16.632929    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24ca21f3-521c-4bf8-80b4-1c0a7757a0bd-openshift-service-ca\") pod \"maas-api-6bf6d56f9f-9zkqk\" (UID: \"24ca21f3-521c-4bf8-80b4-1c0a7757a0bd\") " pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-9zkqk"
Oct 31 23:43:16.633110 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:16.633021    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxw9d\" (UniqueName: \"kubernetes.io/projected/24ca21f3-521c-4bf8-80b4-1c0a7757a0bd-kube-api-access-hxw9d\") pod \"maas-api-6bf6d56f9f-9zkqk\" (UID: \"24ca21f3-521c-4bf8-80b4-1c0a7757a0bd\") " pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-9zkqk"
Oct 31 23:43:16.633211 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:16.633121    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24ca21f3-521c-4bf8-80b4-1c0a7757a0bd-metrics-certs\") pod \"maas-api-6bf6d56f9f-9zkqk\" (UID: \"24ca21f3-521c-4bf8-80b4-1c0a7757a0bd\") " pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-9zkqk"
Oct 31 23:43:16.733766 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:16.733732    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24ca21f3-521c-4bf8-80b4-1c0a7757a0bd-openshift-service-ca\") pod \"maas-api-6bf6d56f9f-9zkqk\" (UID: \"24ca21f3-521c-4bf8-80b4-1c0a7757a0bd\") " pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-9zkqk"
Oct 31 23:43:16.733766 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:16.733770    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxw9d\" (UniqueName: \"kubernetes.io/projected/24ca21f3-521c-4bf8-80b4-1c0a7757a0bd-kube-api-access-hxw9d\") pod \"maas-api-6bf6d56f9f-9zkqk\" (UID: \"24ca21f3-521c-4bf8-80b4-1c0a7757a0bd\") " pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-9zkqk"
Oct 31 23:43:16.734021 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:16.733803    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24ca21f3-521c-4bf8-80b4-1c0a7757a0bd-metrics-certs\") pod \"maas-api-6bf6d56f9f-9zkqk\" (UID: \"24ca21f3-521c-4bf8-80b4-1c0a7757a0bd\") " pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-9zkqk"
Oct 31 23:43:16.734021 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:16.733839    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24ca21f3-521c-4bf8-80b4-1c0a7757a0bd-maas-api-tls\") pod \"maas-api-6bf6d56f9f-9zkqk\" (UID: \"24ca21f3-521c-4bf8-80b4-1c0a7757a0bd\") " pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-9zkqk"
Oct 31 23:43:16.734498 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:16.734475    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24ca21f3-521c-4bf8-80b4-1c0a7757a0bd-openshift-service-ca\") pod \"maas-api-6bf6d56f9f-9zkqk\" (UID: \"24ca21f3-521c-4bf8-80b4-1c0a7757a0bd\") " pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-9zkqk"
Oct 31 23:43:16.736429 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:16.736383    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24ca21f3-521c-4bf8-80b4-1c0a7757a0bd-metrics-certs\") pod \"maas-api-6bf6d56f9f-9zkqk\" (UID: \"24ca21f3-521c-4bf8-80b4-1c0a7757a0bd\") " pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-9zkqk"
Oct 31 23:43:16.736532 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:16.736387    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24ca21f3-521c-4bf8-80b4-1c0a7757a0bd-maas-api-tls\") pod \"maas-api-6bf6d56f9f-9zkqk\" (UID: \"24ca21f3-521c-4bf8-80b4-1c0a7757a0bd\") " pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-9zkqk"
Oct 31 23:43:16.745010 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:16.744989    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxw9d\" (UniqueName: \"kubernetes.io/projected/24ca21f3-521c-4bf8-80b4-1c0a7757a0bd-kube-api-access-hxw9d\") pod \"maas-api-6bf6d56f9f-9zkqk\" (UID: \"24ca21f3-521c-4bf8-80b4-1c0a7757a0bd\") " pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-9zkqk"
Oct 31 23:43:16.842663 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:16.842587    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-9zkqk"
Oct 31 23:43:17.117595 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:17.117318    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6bf6d56f9f-9zkqk"]
Oct 31 23:43:17.120278 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:43:17.120251    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ca21f3_521c_4bf8_80b4_1c0a7757a0bd.slice/crio-1dc389707f6adca2f808f10f9a180f4b6683535bbeec6cb8a0c7dcff78db46fd WatchSource:0}: Error finding container 1dc389707f6adca2f808f10f9a180f4b6683535bbeec6cb8a0c7dcff78db46fd: Status 404 returned error can't find the container with id 1dc389707f6adca2f808f10f9a180f4b6683535bbeec6cb8a0c7dcff78db46fd
Oct 31 23:43:17.279448 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:17.279406    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-9zkqk" event={"ID":"24ca21f3-521c-4bf8-80b4-1c0a7757a0bd","Type":"ContainerStarted","Data":"1dc389707f6adca2f808f10f9a180f4b6683535bbeec6cb8a0c7dcff78db46fd"}
Oct 31 23:43:18.284763 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:18.284726    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-9zkqk" event={"ID":"24ca21f3-521c-4bf8-80b4-1c0a7757a0bd","Type":"ContainerStarted","Data":"0878bbf411a9cf8751f9dd902e5f0790b0af8c5203365f959b893aaedd1525a7"}
Oct 31 23:43:18.285218 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:18.284839    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-9zkqk"
Oct 31 23:43:18.303509 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:18.303457    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-9zkqk" podStartSLOduration=1.8589522349999998 podStartE2EDuration="2.303445284s" totalImagesPullingTime="444.493049ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:43:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:43:18.302846044 +0000 UTC m=+3217.050035542" watchObservedRunningTime="2026-10-31 23:43:18.303445284 +0000 UTC m=+3217.050634788"
Oct 31 23:43:24.294860 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:24.294831    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-9zkqk"
Oct 31 23:43:24.342758 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:24.342719    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6784f6bdbc-jgktm"]
Oct 31 23:43:24.342991 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:24.342953    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-jgktm" podUID="06b93593-507a-4653-9fc8-59dacbdfd71f" containerName="maas-api" containerID="cri-o://e5689157879eaca6989c1e8f95d302b3829f3cbf9fa23e6855c71cbb53719a2e" gracePeriod=30
Oct 31 23:43:25.580477 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:25.580449    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-jgktm"
Oct 31 23:43:25.603289 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:25.603261    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06b93593-507a-4653-9fc8-59dacbdfd71f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06b93593-507a-4653-9fc8-59dacbdfd71f-openshift-service-ca\") pod \"06b93593-507a-4653-9fc8-59dacbdfd71f\" (UID: \"06b93593-507a-4653-9fc8-59dacbdfd71f\") "
Oct 31 23:43:25.603662 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:25.603342    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06b93593-507a-4653-9fc8-59dacbdfd71f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06b93593-507a-4653-9fc8-59dacbdfd71f-maas-api-tls\") pod \"06b93593-507a-4653-9fc8-59dacbdfd71f\" (UID: \"06b93593-507a-4653-9fc8-59dacbdfd71f\") "
Oct 31 23:43:25.603662 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:25.603455    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06b93593-507a-4653-9fc8-59dacbdfd71f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/06b93593-507a-4653-9fc8-59dacbdfd71f-metrics-certs\") pod \"06b93593-507a-4653-9fc8-59dacbdfd71f\" (UID: \"06b93593-507a-4653-9fc8-59dacbdfd71f\") "
Oct 31 23:43:25.603662 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:25.603602    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06b93593-507a-4653-9fc8-59dacbdfd71f-kube-api-access-2btjt\" (UniqueName: \"kubernetes.io/projected/06b93593-507a-4653-9fc8-59dacbdfd71f-kube-api-access-2btjt\") pod \"06b93593-507a-4653-9fc8-59dacbdfd71f\" (UID: \"06b93593-507a-4653-9fc8-59dacbdfd71f\") "
Oct 31 23:43:25.603983 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:25.603707    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06b93593-507a-4653-9fc8-59dacbdfd71f-openshift-service-ca" pod "06b93593-507a-4653-9fc8-59dacbdfd71f" (UID: "06b93593-507a-4653-9fc8-59dacbdfd71f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:43:25.604539 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:25.604503    4301 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06b93593-507a-4653-9fc8-59dacbdfd71f-openshift-service-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:43:25.606075 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:25.606052    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06b93593-507a-4653-9fc8-59dacbdfd71f-metrics-certs" pod "06b93593-507a-4653-9fc8-59dacbdfd71f" (UID: "06b93593-507a-4653-9fc8-59dacbdfd71f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:43:25.606266 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:25.606224    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06b93593-507a-4653-9fc8-59dacbdfd71f-maas-api-tls" pod "06b93593-507a-4653-9fc8-59dacbdfd71f" (UID: "06b93593-507a-4653-9fc8-59dacbdfd71f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:43:25.606266 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:25.606248    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06b93593-507a-4653-9fc8-59dacbdfd71f-kube-api-access-2btjt" pod "06b93593-507a-4653-9fc8-59dacbdfd71f" (UID: "06b93593-507a-4653-9fc8-59dacbdfd71f"). InnerVolumeSpecName "kube-api-access-2btjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:43:25.705250 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:25.705175    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06b93593-507a-4653-9fc8-59dacbdfd71f-maas-api-tls\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:43:25.705250 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:25.705201    4301 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06b93593-507a-4653-9fc8-59dacbdfd71f-metrics-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:43:25.705250 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:25.705211    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2btjt\" (UniqueName: \"kubernetes.io/projected/06b93593-507a-4653-9fc8-59dacbdfd71f-kube-api-access-2btjt\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:43:26.319175 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:26.319140    4301 generic.go:356] "Generic (PLEG): container finished" podID="06b93593-507a-4653-9fc8-59dacbdfd71f" containerID="e5689157879eaca6989c1e8f95d302b3829f3cbf9fa23e6855c71cbb53719a2e" exitCode=0
Oct 31 23:43:26.319380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:26.319188    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-jgktm"
Oct 31 23:43:26.319380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:26.319227    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-jgktm" event={"ID":"06b93593-507a-4653-9fc8-59dacbdfd71f","Type":"ContainerDied","Data":"e5689157879eaca6989c1e8f95d302b3829f3cbf9fa23e6855c71cbb53719a2e"}
Oct 31 23:43:26.319380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:26.319280    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-jgktm" event={"ID":"06b93593-507a-4653-9fc8-59dacbdfd71f","Type":"ContainerDied","Data":"c39f9a3c225ac31e61650a285e956e51c42116394b58924949a21a00a744d499"}
Oct 31 23:43:26.319380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:26.319302    4301 scope.go:122] "RemoveContainer" containerID="e5689157879eaca6989c1e8f95d302b3829f3cbf9fa23e6855c71cbb53719a2e"
Oct 31 23:43:26.330092 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:26.330003    4301 scope.go:122] "RemoveContainer" containerID="e5689157879eaca6989c1e8f95d302b3829f3cbf9fa23e6855c71cbb53719a2e"
Oct 31 23:43:26.330552 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:43:26.330530    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5689157879eaca6989c1e8f95d302b3829f3cbf9fa23e6855c71cbb53719a2e\": container with ID starting with e5689157879eaca6989c1e8f95d302b3829f3cbf9fa23e6855c71cbb53719a2e not found: ID does not exist" containerID="e5689157879eaca6989c1e8f95d302b3829f3cbf9fa23e6855c71cbb53719a2e"
Oct 31 23:43:26.330660 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:26.330558    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5689157879eaca6989c1e8f95d302b3829f3cbf9fa23e6855c71cbb53719a2e"} err="failed to get container status \"e5689157879eaca6989c1e8f95d302b3829f3cbf9fa23e6855c71cbb53719a2e\": rpc error: code = NotFound desc = could not find container \"e5689157879eaca6989c1e8f95d302b3829f3cbf9fa23e6855c71cbb53719a2e\": container with ID starting with e5689157879eaca6989c1e8f95d302b3829f3cbf9fa23e6855c71cbb53719a2e not found: ID does not exist"
Oct 31 23:43:26.342200 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:26.342175    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6784f6bdbc-jgktm"]
Oct 31 23:43:26.347877 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:26.347857    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6784f6bdbc-jgktm"]
Oct 31 23:43:27.402017 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:27.401983    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06b93593-507a-4653-9fc8-59dacbdfd71f" path="/var/lib/kubelet/pods/06b93593-507a-4653-9fc8-59dacbdfd71f/volumes"
Oct 31 23:43:27.914987 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:27.914954    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm"]
Oct 31 23:43:27.915309 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:27.915290    4301 cpu_manager.go:414] "containerMap: removing container" podUID="06b93593-507a-4653-9fc8-59dacbdfd71f" containerName="maas-api"
Oct 31 23:43:27.915424 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:27.915310    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06b93593-507a-4653-9fc8-59dacbdfd71f" containerName="maas-api"
Oct 31 23:43:27.915493 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:27.915445    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="06b93593-507a-4653-9fc8-59dacbdfd71f" containerName="maas-api"
Oct 31 23:43:27.919869 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:27.919846    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm"
Oct 31 23:43:27.930612 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:27.930590    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm"]
Oct 31 23:43:27.979130 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:27.979103    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3d08af8f-serving-cert\""
Oct 31 23:43:28.026647 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.026615    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45df8720-ddc1-42e1-aa96-e2eefa7d6e8b-metrics-certs\") pod \"maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm\" (UID: \"45df8720-ddc1-42e1-aa96-e2eefa7d6e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm"
Oct 31 23:43:28.026840 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.026668    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45df8720-ddc1-42e1-aa96-e2eefa7d6e8b-maas-api-tls\") pod \"maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm\" (UID: \"45df8720-ddc1-42e1-aa96-e2eefa7d6e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm"
Oct 31 23:43:28.026840 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.026743    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf9v5\" (UniqueName: \"kubernetes.io/projected/45df8720-ddc1-42e1-aa96-e2eefa7d6e8b-kube-api-access-gf9v5\") pod \"maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm\" (UID: \"45df8720-ddc1-42e1-aa96-e2eefa7d6e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm"
Oct 31 23:43:28.026840 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.026769    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45df8720-ddc1-42e1-aa96-e2eefa7d6e8b-openshift-service-ca\") pod \"maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm\" (UID: \"45df8720-ddc1-42e1-aa96-e2eefa7d6e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm"
Oct 31 23:43:28.028935 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.028906    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3d08af8f-75d47f56bc-w6474"]
Oct 31 23:43:28.032183 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.032167    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3d08af8f-75d47f56bc-w6474"
Oct 31 23:43:28.037960 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.037940    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3d08af8f\""
Oct 31 23:43:28.044538 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.044517    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3d08af8f-75d47f56bc-w6474"]
Oct 31 23:43:28.055561 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.055541    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3d08af8f-dockercfg-wv2tk\""
Oct 31 23:43:28.069314 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.069288    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3d08af8f-5758bd6c66-2h767"]
Oct 31 23:43:28.072583 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.072568    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3d08af8f-5758bd6c66-2h767"
Oct 31 23:43:28.087474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.087449    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3d08af8f-5758bd6c66-2h767"]
Oct 31 23:43:28.127703 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.127611    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45df8720-ddc1-42e1-aa96-e2eefa7d6e8b-metrics-certs\") pod \"maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm\" (UID: \"45df8720-ddc1-42e1-aa96-e2eefa7d6e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm"
Oct 31 23:43:28.127703 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.127672    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw2vp\" (UniqueName: \"kubernetes.io/projected/1a43534d-982b-461b-a1bb-520cc0ca59ff-kube-api-access-lw2vp\") pod \"payload-processing-e2e-ait-3d08af8f-5758bd6c66-2h767\" (UID: \"1a43534d-982b-461b-a1bb-520cc0ca59ff\") " pod="openshift-ingress/payload-processing-e2e-ait-3d08af8f-5758bd6c66-2h767"
Oct 31 23:43:28.128023 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.127721    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45df8720-ddc1-42e1-aa96-e2eefa7d6e8b-maas-api-tls\") pod \"maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm\" (UID: \"45df8720-ddc1-42e1-aa96-e2eefa7d6e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm"
Oct 31 23:43:28.128023 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.127749    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf9v5\" (UniqueName: \"kubernetes.io/projected/45df8720-ddc1-42e1-aa96-e2eefa7d6e8b-kube-api-access-gf9v5\") pod \"maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm\" (UID: \"45df8720-ddc1-42e1-aa96-e2eefa7d6e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm"
Oct 31 23:43:28.128023 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.127780    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45df8720-ddc1-42e1-aa96-e2eefa7d6e8b-openshift-service-ca\") pod \"maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm\" (UID: \"45df8720-ddc1-42e1-aa96-e2eefa7d6e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm"
Oct 31 23:43:28.128023 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.127841    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52d0ad8-6738-46a3-a64f-9824375d79f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3d08af8f-75d47f56bc-w6474\" (UID: \"e52d0ad8-6738-46a3-a64f-9824375d79f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3d08af8f-75d47f56bc-w6474"
Oct 31 23:43:28.128023 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.127891    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a43534d-982b-461b-a1bb-520cc0ca59ff-plugins-config-volume\") pod \"payload-processing-e2e-ait-3d08af8f-5758bd6c66-2h767\" (UID: \"1a43534d-982b-461b-a1bb-520cc0ca59ff\") " pod="openshift-ingress/payload-processing-e2e-ait-3d08af8f-5758bd6c66-2h767"
Oct 31 23:43:28.128023 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.127923    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jj24\" (UniqueName: \"kubernetes.io/projected/e52d0ad8-6738-46a3-a64f-9824375d79f3-kube-api-access-6jj24\") pod \"payload-pre-processing-e2e-ait-3d08af8f-75d47f56bc-w6474\" (UID: \"e52d0ad8-6738-46a3-a64f-9824375d79f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3d08af8f-75d47f56bc-w6474"
Oct 31 23:43:28.128681 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:43:28.128546    4301 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-3d08af8f-serving-cert: secret "maas-api-e2e-ait-3d08af8f-serving-cert" not found
Oct 31 23:43:28.128681 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:43:28.128632    4301 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45df8720-ddc1-42e1-aa96-e2eefa7d6e8b-maas-api-tls podName:45df8720-ddc1-42e1-aa96-e2eefa7d6e8b nodeName:}" failed. No retries permitted until 2026-10-31 23:43:28.628592634 +0000 UTC m=+3227.375782132 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/45df8720-ddc1-42e1-aa96-e2eefa7d6e8b-maas-api-tls") pod "maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm" (UID: "45df8720-ddc1-42e1-aa96-e2eefa7d6e8b") : secret "maas-api-e2e-ait-3d08af8f-serving-cert" not found
Oct 31 23:43:28.129249 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.129228    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45df8720-ddc1-42e1-aa96-e2eefa7d6e8b-openshift-service-ca\") pod \"maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm\" (UID: \"45df8720-ddc1-42e1-aa96-e2eefa7d6e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm"
Oct 31 23:43:28.130801 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.130770    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45df8720-ddc1-42e1-aa96-e2eefa7d6e8b-metrics-certs\") pod \"maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm\" (UID: \"45df8720-ddc1-42e1-aa96-e2eefa7d6e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm"
Oct 31 23:43:28.147094 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.147064    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf9v5\" (UniqueName: \"kubernetes.io/projected/45df8720-ddc1-42e1-aa96-e2eefa7d6e8b-kube-api-access-gf9v5\") pod \"maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm\" (UID: \"45df8720-ddc1-42e1-aa96-e2eefa7d6e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm"
Oct 31 23:43:28.229470 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.229387    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52d0ad8-6738-46a3-a64f-9824375d79f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3d08af8f-75d47f56bc-w6474\" (UID: \"e52d0ad8-6738-46a3-a64f-9824375d79f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3d08af8f-75d47f56bc-w6474"
Oct 31 23:43:28.229470 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.229460    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a43534d-982b-461b-a1bb-520cc0ca59ff-plugins-config-volume\") pod \"payload-processing-e2e-ait-3d08af8f-5758bd6c66-2h767\" (UID: \"1a43534d-982b-461b-a1bb-520cc0ca59ff\") " pod="openshift-ingress/payload-processing-e2e-ait-3d08af8f-5758bd6c66-2h767"
Oct 31 23:43:28.229679 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.229492    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jj24\" (UniqueName: \"kubernetes.io/projected/e52d0ad8-6738-46a3-a64f-9824375d79f3-kube-api-access-6jj24\") pod \"payload-pre-processing-e2e-ait-3d08af8f-75d47f56bc-w6474\" (UID: \"e52d0ad8-6738-46a3-a64f-9824375d79f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3d08af8f-75d47f56bc-w6474"
Oct 31 23:43:28.229679 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.229565    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw2vp\" (UniqueName: \"kubernetes.io/projected/1a43534d-982b-461b-a1bb-520cc0ca59ff-kube-api-access-lw2vp\") pod \"payload-processing-e2e-ait-3d08af8f-5758bd6c66-2h767\" (UID: \"1a43534d-982b-461b-a1bb-520cc0ca59ff\") " pod="openshift-ingress/payload-processing-e2e-ait-3d08af8f-5758bd6c66-2h767"
Oct 31 23:43:28.230177 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.230156    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52d0ad8-6738-46a3-a64f-9824375d79f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3d08af8f-75d47f56bc-w6474\" (UID: \"e52d0ad8-6738-46a3-a64f-9824375d79f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3d08af8f-75d47f56bc-w6474"
Oct 31 23:43:28.230217 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.230173    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a43534d-982b-461b-a1bb-520cc0ca59ff-plugins-config-volume\") pod \"payload-processing-e2e-ait-3d08af8f-5758bd6c66-2h767\" (UID: \"1a43534d-982b-461b-a1bb-520cc0ca59ff\") " pod="openshift-ingress/payload-processing-e2e-ait-3d08af8f-5758bd6c66-2h767"
Oct 31 23:43:28.241807 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.241783    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw2vp\" (UniqueName: \"kubernetes.io/projected/1a43534d-982b-461b-a1bb-520cc0ca59ff-kube-api-access-lw2vp\") pod \"payload-processing-e2e-ait-3d08af8f-5758bd6c66-2h767\" (UID: \"1a43534d-982b-461b-a1bb-520cc0ca59ff\") " pod="openshift-ingress/payload-processing-e2e-ait-3d08af8f-5758bd6c66-2h767"
Oct 31 23:43:28.241807 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.241795    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jj24\" (UniqueName: \"kubernetes.io/projected/e52d0ad8-6738-46a3-a64f-9824375d79f3-kube-api-access-6jj24\") pod \"payload-pre-processing-e2e-ait-3d08af8f-75d47f56bc-w6474\" (UID: \"e52d0ad8-6738-46a3-a64f-9824375d79f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3d08af8f-75d47f56bc-w6474"
Oct 31 23:43:28.345326 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.345295    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3d08af8f-75d47f56bc-w6474"
Oct 31 23:43:28.384663 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.384622    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3d08af8f-5758bd6c66-2h767"
Oct 31 23:43:28.632326 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.632254    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3d08af8f-75d47f56bc-w6474"]
Oct 31 23:43:28.632785 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.632576    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45df8720-ddc1-42e1-aa96-e2eefa7d6e8b-maas-api-tls\") pod \"maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm\" (UID: \"45df8720-ddc1-42e1-aa96-e2eefa7d6e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm"
Oct 31 23:43:28.634991 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.634965    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45df8720-ddc1-42e1-aa96-e2eefa7d6e8b-maas-api-tls\") pod \"maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm\" (UID: \"45df8720-ddc1-42e1-aa96-e2eefa7d6e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm"
Oct 31 23:43:28.635123 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:43:28.634992    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode52d0ad8_6738_46a3_a64f_9824375d79f3.slice/crio-ddb426c72a5e098678ac81515b6ee67231319a761125148cafe0b1eff16513d4 WatchSource:0}: Error finding container ddb426c72a5e098678ac81515b6ee67231319a761125148cafe0b1eff16513d4: Status 404 returned error can't find the container with id ddb426c72a5e098678ac81515b6ee67231319a761125148cafe0b1eff16513d4
Oct 31 23:43:28.670896 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.670875    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3d08af8f-5758bd6c66-2h767"]
Oct 31 23:43:28.673683 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:43:28.673659    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a43534d_982b_461b_a1bb_520cc0ca59ff.slice/crio-d395a5b503b65da95845d778274e12959fb5e84002dd639ae27568cab151ac36 WatchSource:0}: Error finding container d395a5b503b65da95845d778274e12959fb5e84002dd639ae27568cab151ac36: Status 404 returned error can't find the container with id d395a5b503b65da95845d778274e12959fb5e84002dd639ae27568cab151ac36
Oct 31 23:43:28.834906 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:28.834870    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm"
Oct 31 23:43:29.107842 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:29.107810    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm"]
Oct 31 23:43:29.111009 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:43:29.110981    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45df8720_ddc1_42e1_aa96_e2eefa7d6e8b.slice/crio-c847e0b7822b491f60d9a9e33a555703ac7e2a5e0086db28f5c94b349d40be52 WatchSource:0}: Error finding container c847e0b7822b491f60d9a9e33a555703ac7e2a5e0086db28f5c94b349d40be52: Status 404 returned error can't find the container with id c847e0b7822b491f60d9a9e33a555703ac7e2a5e0086db28f5c94b349d40be52
Oct 31 23:43:29.335865 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:29.335769    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3d08af8f-5758bd6c66-2h767" event={"ID":"1a43534d-982b-461b-a1bb-520cc0ca59ff","Type":"ContainerStarted","Data":"d1bb54647044b3dbf94f62820e822181bea871a3347dcd648aa7fc604fc9c389"}
Oct 31 23:43:29.335865 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:29.335823    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3d08af8f-5758bd6c66-2h767" event={"ID":"1a43534d-982b-461b-a1bb-520cc0ca59ff","Type":"ContainerStarted","Data":"d395a5b503b65da95845d778274e12959fb5e84002dd639ae27568cab151ac36"}
Oct 31 23:43:29.335865 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:29.335860    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-3d08af8f-5758bd6c66-2h767"
Oct 31 23:43:29.337421 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:29.337393    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3d08af8f-75d47f56bc-w6474" event={"ID":"e52d0ad8-6738-46a3-a64f-9824375d79f3","Type":"ContainerStarted","Data":"9354e30efda48ca9c5ef5e9938283fae6c5ff6022752852eca9a096d09ae508d"}
Oct 31 23:43:29.337500 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:29.337444    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3d08af8f-75d47f56bc-w6474" event={"ID":"e52d0ad8-6738-46a3-a64f-9824375d79f3","Type":"ContainerStarted","Data":"ddb426c72a5e098678ac81515b6ee67231319a761125148cafe0b1eff16513d4"}
Oct 31 23:43:29.337500 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:29.337475    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3d08af8f-75d47f56bc-w6474"
Oct 31 23:43:29.338664 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:29.338642    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm" event={"ID":"45df8720-ddc1-42e1-aa96-e2eefa7d6e8b","Type":"ContainerStarted","Data":"c847e0b7822b491f60d9a9e33a555703ac7e2a5e0086db28f5c94b349d40be52"}
Oct 31 23:43:29.354957 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:29.354922    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-3d08af8f-5758bd6c66-2h767" podStartSLOduration=1.3549115170000001 podStartE2EDuration="1.354911517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:43:29.354307333 +0000 UTC m=+3228.101496837" watchObservedRunningTime="2026-10-31 23:43:29.354911517 +0000 UTC m=+3228.102101019"
Oct 31 23:43:29.372763 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:29.372720    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-3d08af8f-75d47f56bc-w6474" podStartSLOduration=1.372708894 podStartE2EDuration="1.372708894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:43:29.371280434 +0000 UTC m=+3228.118469939" watchObservedRunningTime="2026-10-31 23:43:29.372708894 +0000 UTC m=+3228.119898403"
Oct 31 23:43:30.345117 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:30.345073    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm" event={"ID":"45df8720-ddc1-42e1-aa96-e2eefa7d6e8b","Type":"ContainerStarted","Data":"2e61c35427ec66d86918f1d4c7c3e4ccae0c8c3ce1b3edf0054ef864daccb737"}
Oct 31 23:43:30.345663 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:30.345186    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm"
Oct 31 23:43:30.366063 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:30.366012    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm" podStartSLOduration=2.996511254 podStartE2EDuration="3.365996151s" totalImagesPullingTime="369.484897ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:43:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:43:30.365088378 +0000 UTC m=+3229.112277894" watchObservedRunningTime="2026-10-31 23:43:30.365996151 +0000 UTC m=+3229.113185692"
Oct 31 23:43:30.408518 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:30.408486    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:43:32.544705 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:43:32.544666    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:43:36.354714 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:36.354686    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm"
Oct 31 23:43:40.348675 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:40.348640    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3d08af8f-75d47f56bc-w6474"
Oct 31 23:43:40.349224 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:40.348705    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-3d08af8f-5758bd6c66-2h767"
Oct 31 23:43:52.796507 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:52.796461    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3d08af8f-5758bd6c66-2h767"]
Oct 31 23:43:52.806836 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:52.806804    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3d08af8f-75d47f56bc-w6474"]
Oct 31 23:43:53.504791 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:53.504755    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm"]
Oct 31 23:43:53.505068 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:53.505027    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm" podUID="45df8720-ddc1-42e1-aa96-e2eefa7d6e8b" containerName="maas-api" containerID="cri-o://2e61c35427ec66d86918f1d4c7c3e4ccae0c8c3ce1b3edf0054ef864daccb737" gracePeriod=30
Oct 31 23:43:53.758298 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:53.758231    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm"
Oct 31 23:43:53.867089 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:53.867053    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45df8720-ddc1-42e1-aa96-e2eefa7d6e8b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45df8720-ddc1-42e1-aa96-e2eefa7d6e8b-openshift-service-ca\") pod \"45df8720-ddc1-42e1-aa96-e2eefa7d6e8b\" (UID: \"45df8720-ddc1-42e1-aa96-e2eefa7d6e8b\") "
Oct 31 23:43:53.867541 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:53.867122    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45df8720-ddc1-42e1-aa96-e2eefa7d6e8b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45df8720-ddc1-42e1-aa96-e2eefa7d6e8b-maas-api-tls\") pod \"45df8720-ddc1-42e1-aa96-e2eefa7d6e8b\" (UID: \"45df8720-ddc1-42e1-aa96-e2eefa7d6e8b\") "
Oct 31 23:43:53.867541 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:53.867169    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45df8720-ddc1-42e1-aa96-e2eefa7d6e8b-kube-api-access-gf9v5\" (UniqueName: \"kubernetes.io/projected/45df8720-ddc1-42e1-aa96-e2eefa7d6e8b-kube-api-access-gf9v5\") pod \"45df8720-ddc1-42e1-aa96-e2eefa7d6e8b\" (UID: \"45df8720-ddc1-42e1-aa96-e2eefa7d6e8b\") "
Oct 31 23:43:53.867541 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:53.867251    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45df8720-ddc1-42e1-aa96-e2eefa7d6e8b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/45df8720-ddc1-42e1-aa96-e2eefa7d6e8b-metrics-certs\") pod \"45df8720-ddc1-42e1-aa96-e2eefa7d6e8b\" (UID: \"45df8720-ddc1-42e1-aa96-e2eefa7d6e8b\") "
Oct 31 23:43:53.867541 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:53.867520    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45df8720-ddc1-42e1-aa96-e2eefa7d6e8b-openshift-service-ca" pod "45df8720-ddc1-42e1-aa96-e2eefa7d6e8b" (UID: "45df8720-ddc1-42e1-aa96-e2eefa7d6e8b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:43:53.869501 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:53.869473    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45df8720-ddc1-42e1-aa96-e2eefa7d6e8b-kube-api-access-gf9v5" pod "45df8720-ddc1-42e1-aa96-e2eefa7d6e8b" (UID: "45df8720-ddc1-42e1-aa96-e2eefa7d6e8b"). InnerVolumeSpecName "kube-api-access-gf9v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:43:53.869501 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:53.869483    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45df8720-ddc1-42e1-aa96-e2eefa7d6e8b-metrics-certs" pod "45df8720-ddc1-42e1-aa96-e2eefa7d6e8b" (UID: "45df8720-ddc1-42e1-aa96-e2eefa7d6e8b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:43:53.869672 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:53.869500    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45df8720-ddc1-42e1-aa96-e2eefa7d6e8b-maas-api-tls" pod "45df8720-ddc1-42e1-aa96-e2eefa7d6e8b" (UID: "45df8720-ddc1-42e1-aa96-e2eefa7d6e8b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 23:43:53.968311 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:53.968280    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45df8720-ddc1-42e1-aa96-e2eefa7d6e8b-maas-api-tls\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:43:53.968311 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:53.968306    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf9v5\" (UniqueName: \"kubernetes.io/projected/45df8720-ddc1-42e1-aa96-e2eefa7d6e8b-kube-api-access-gf9v5\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:43:53.968311 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:53.968319    4301 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45df8720-ddc1-42e1-aa96-e2eefa7d6e8b-metrics-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:43:53.968588 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:53.968328    4301 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45df8720-ddc1-42e1-aa96-e2eefa7d6e8b-openshift-service-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:43:54.447438 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:54.447399    4301 generic.go:356] "Generic (PLEG): container finished" podID="45df8720-ddc1-42e1-aa96-e2eefa7d6e8b" containerID="2e61c35427ec66d86918f1d4c7c3e4ccae0c8c3ce1b3edf0054ef864daccb737" exitCode=0
Oct 31 23:43:54.447616 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:54.447449    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm"
Oct 31 23:43:54.447616 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:54.447484    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm" event={"ID":"45df8720-ddc1-42e1-aa96-e2eefa7d6e8b","Type":"ContainerDied","Data":"2e61c35427ec66d86918f1d4c7c3e4ccae0c8c3ce1b3edf0054ef864daccb737"}
Oct 31 23:43:54.447616 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:54.447539    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm" event={"ID":"45df8720-ddc1-42e1-aa96-e2eefa7d6e8b","Type":"ContainerDied","Data":"c847e0b7822b491f60d9a9e33a555703ac7e2a5e0086db28f5c94b349d40be52"}
Oct 31 23:43:54.447616 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:54.447563    4301 scope.go:122] "RemoveContainer" containerID="2e61c35427ec66d86918f1d4c7c3e4ccae0c8c3ce1b3edf0054ef864daccb737"
Oct 31 23:43:54.456947 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:54.456926    4301 scope.go:122] "RemoveContainer" containerID="2e61c35427ec66d86918f1d4c7c3e4ccae0c8c3ce1b3edf0054ef864daccb737"
Oct 31 23:43:54.457230 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:43:54.457213    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e61c35427ec66d86918f1d4c7c3e4ccae0c8c3ce1b3edf0054ef864daccb737\": container with ID starting with 2e61c35427ec66d86918f1d4c7c3e4ccae0c8c3ce1b3edf0054ef864daccb737 not found: ID does not exist" containerID="2e61c35427ec66d86918f1d4c7c3e4ccae0c8c3ce1b3edf0054ef864daccb737"
Oct 31 23:43:54.457271 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:54.457241    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e61c35427ec66d86918f1d4c7c3e4ccae0c8c3ce1b3edf0054ef864daccb737"} err="failed to get container status \"2e61c35427ec66d86918f1d4c7c3e4ccae0c8c3ce1b3edf0054ef864daccb737\": rpc error: code = NotFound desc = could not find container \"2e61c35427ec66d86918f1d4c7c3e4ccae0c8c3ce1b3edf0054ef864daccb737\": container with ID starting with 2e61c35427ec66d86918f1d4c7c3e4ccae0c8c3ce1b3edf0054ef864daccb737 not found: ID does not exist"
Oct 31 23:43:54.467430 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:54.467407    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm"]
Oct 31 23:43:54.471404 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:54.471381    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3d08af8f-64cc794cc5-jvrzm"]
Oct 31 23:43:54.927490 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:54.927460    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:43:55.402399 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:55.402351    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45df8720-ddc1-42e1-aa96-e2eefa7d6e8b" path="/var/lib/kubelet/pods/45df8720-ddc1-42e1-aa96-e2eefa7d6e8b/volumes"
Oct 31 23:43:57.797752 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:57.797689    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-3d08af8f-5758bd6c66-2h767" podUID="1a43534d-982b-461b-a1bb-520cc0ca59ff" containerName="payload-processing" containerID="cri-o://d1bb54647044b3dbf94f62820e822181bea871a3347dcd648aa7fc604fc9c389" gracePeriod=25
Oct 31 23:43:57.807341 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:57.807311    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-3d08af8f-75d47f56bc-w6474" podUID="e52d0ad8-6738-46a3-a64f-9824375d79f3" containerName="payload-pre-processing" containerID="cri-o://9354e30efda48ca9c5ef5e9938283fae6c5ff6022752852eca9a096d09ae508d" gracePeriod=25
Oct 31 23:43:58.042341 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:58.042318    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3d08af8f-5758bd6c66-2h767"
Oct 31 23:43:58.072898 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:58.072875    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3d08af8f-75d47f56bc-w6474"
Oct 31 23:43:58.205371 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:58.205333    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a43534d-982b-461b-a1bb-520cc0ca59ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a43534d-982b-461b-a1bb-520cc0ca59ff-plugins-config-volume\") pod \"1a43534d-982b-461b-a1bb-520cc0ca59ff\" (UID: \"1a43534d-982b-461b-a1bb-520cc0ca59ff\") "
Oct 31 23:43:58.205563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:58.205386    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e52d0ad8-6738-46a3-a64f-9824375d79f3-kube-api-access-6jj24\" (UniqueName: \"kubernetes.io/projected/e52d0ad8-6738-46a3-a64f-9824375d79f3-kube-api-access-6jj24\") pod \"e52d0ad8-6738-46a3-a64f-9824375d79f3\" (UID: \"e52d0ad8-6738-46a3-a64f-9824375d79f3\") "
Oct 31 23:43:58.205563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:58.205407    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a43534d-982b-461b-a1bb-520cc0ca59ff-kube-api-access-lw2vp\" (UniqueName: \"kubernetes.io/projected/1a43534d-982b-461b-a1bb-520cc0ca59ff-kube-api-access-lw2vp\") pod \"1a43534d-982b-461b-a1bb-520cc0ca59ff\" (UID: \"1a43534d-982b-461b-a1bb-520cc0ca59ff\") "
Oct 31 23:43:58.205563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:58.205427    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e52d0ad8-6738-46a3-a64f-9824375d79f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52d0ad8-6738-46a3-a64f-9824375d79f3-plugins-config-volume\") pod \"e52d0ad8-6738-46a3-a64f-9824375d79f3\" (UID: \"e52d0ad8-6738-46a3-a64f-9824375d79f3\") "
Oct 31 23:43:58.205843 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:58.205811    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a43534d-982b-461b-a1bb-520cc0ca59ff-plugins-config-volume" pod "1a43534d-982b-461b-a1bb-520cc0ca59ff" (UID: "1a43534d-982b-461b-a1bb-520cc0ca59ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:43:58.205972 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:58.205886    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e52d0ad8-6738-46a3-a64f-9824375d79f3-plugins-config-volume" pod "e52d0ad8-6738-46a3-a64f-9824375d79f3" (UID: "e52d0ad8-6738-46a3-a64f-9824375d79f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:43:58.207551 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:58.207530    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a43534d-982b-461b-a1bb-520cc0ca59ff-kube-api-access-lw2vp" pod "1a43534d-982b-461b-a1bb-520cc0ca59ff" (UID: "1a43534d-982b-461b-a1bb-520cc0ca59ff"). InnerVolumeSpecName "kube-api-access-lw2vp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:43:58.207624 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:58.207583    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e52d0ad8-6738-46a3-a64f-9824375d79f3-kube-api-access-6jj24" pod "e52d0ad8-6738-46a3-a64f-9824375d79f3" (UID: "e52d0ad8-6738-46a3-a64f-9824375d79f3"). InnerVolumeSpecName "kube-api-access-6jj24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:43:58.306962 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:58.306930    4301 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a43534d-982b-461b-a1bb-520cc0ca59ff-plugins-config-volume\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:43:58.306962 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:58.306956    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jj24\" (UniqueName: \"kubernetes.io/projected/e52d0ad8-6738-46a3-a64f-9824375d79f3-kube-api-access-6jj24\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:43:58.306962 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:58.306965    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw2vp\" (UniqueName: \"kubernetes.io/projected/1a43534d-982b-461b-a1bb-520cc0ca59ff-kube-api-access-lw2vp\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:43:58.307250 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:58.306977    4301 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52d0ad8-6738-46a3-a64f-9824375d79f3-plugins-config-volume\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:43:58.464915 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:58.464878    4301 generic.go:356] "Generic (PLEG): container finished" podID="e52d0ad8-6738-46a3-a64f-9824375d79f3" containerID="9354e30efda48ca9c5ef5e9938283fae6c5ff6022752852eca9a096d09ae508d" exitCode=0
Oct 31 23:43:58.465108 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:58.464928    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3d08af8f-75d47f56bc-w6474"
Oct 31 23:43:58.465108 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:58.464966    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3d08af8f-75d47f56bc-w6474" event={"ID":"e52d0ad8-6738-46a3-a64f-9824375d79f3","Type":"ContainerDied","Data":"9354e30efda48ca9c5ef5e9938283fae6c5ff6022752852eca9a096d09ae508d"}
Oct 31 23:43:58.465108 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:58.465016    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3d08af8f-75d47f56bc-w6474" event={"ID":"e52d0ad8-6738-46a3-a64f-9824375d79f3","Type":"ContainerDied","Data":"ddb426c72a5e098678ac81515b6ee67231319a761125148cafe0b1eff16513d4"}
Oct 31 23:43:58.465108 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:58.465040    4301 scope.go:122] "RemoveContainer" containerID="9354e30efda48ca9c5ef5e9938283fae6c5ff6022752852eca9a096d09ae508d"
Oct 31 23:43:58.466452 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:58.466431    4301 generic.go:356] "Generic (PLEG): container finished" podID="1a43534d-982b-461b-a1bb-520cc0ca59ff" containerID="d1bb54647044b3dbf94f62820e822181bea871a3347dcd648aa7fc604fc9c389" exitCode=0
Oct 31 23:43:58.466581 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:58.466468    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3d08af8f-5758bd6c66-2h767"
Oct 31 23:43:58.466581 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:58.466497    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3d08af8f-5758bd6c66-2h767" event={"ID":"1a43534d-982b-461b-a1bb-520cc0ca59ff","Type":"ContainerDied","Data":"d1bb54647044b3dbf94f62820e822181bea871a3347dcd648aa7fc604fc9c389"}
Oct 31 23:43:58.466581 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:58.466536    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3d08af8f-5758bd6c66-2h767" event={"ID":"1a43534d-982b-461b-a1bb-520cc0ca59ff","Type":"ContainerDied","Data":"d395a5b503b65da95845d778274e12959fb5e84002dd639ae27568cab151ac36"}
Oct 31 23:43:58.474984 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:58.474968    4301 scope.go:122] "RemoveContainer" containerID="9354e30efda48ca9c5ef5e9938283fae6c5ff6022752852eca9a096d09ae508d"
Oct 31 23:43:58.475274 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:43:58.475256    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9354e30efda48ca9c5ef5e9938283fae6c5ff6022752852eca9a096d09ae508d\": container with ID starting with 9354e30efda48ca9c5ef5e9938283fae6c5ff6022752852eca9a096d09ae508d not found: ID does not exist" containerID="9354e30efda48ca9c5ef5e9938283fae6c5ff6022752852eca9a096d09ae508d"
Oct 31 23:43:58.475379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:58.475284    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9354e30efda48ca9c5ef5e9938283fae6c5ff6022752852eca9a096d09ae508d"} err="failed to get container status \"9354e30efda48ca9c5ef5e9938283fae6c5ff6022752852eca9a096d09ae508d\": rpc error: code = NotFound desc = could not find container \"9354e30efda48ca9c5ef5e9938283fae6c5ff6022752852eca9a096d09ae508d\": container with ID starting with 9354e30efda48ca9c5ef5e9938283fae6c5ff6022752852eca9a096d09ae508d not found: ID does not exist"
Oct 31 23:43:58.475379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:58.475308    4301 scope.go:122] "RemoveContainer" containerID="d1bb54647044b3dbf94f62820e822181bea871a3347dcd648aa7fc604fc9c389"
Oct 31 23:43:58.482835 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:58.482819    4301 scope.go:122] "RemoveContainer" containerID="d1bb54647044b3dbf94f62820e822181bea871a3347dcd648aa7fc604fc9c389"
Oct 31 23:43:58.483109 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:43:58.483091    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1bb54647044b3dbf94f62820e822181bea871a3347dcd648aa7fc604fc9c389\": container with ID starting with d1bb54647044b3dbf94f62820e822181bea871a3347dcd648aa7fc604fc9c389 not found: ID does not exist" containerID="d1bb54647044b3dbf94f62820e822181bea871a3347dcd648aa7fc604fc9c389"
Oct 31 23:43:58.483157 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:58.483117    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1bb54647044b3dbf94f62820e822181bea871a3347dcd648aa7fc604fc9c389"} err="failed to get container status \"d1bb54647044b3dbf94f62820e822181bea871a3347dcd648aa7fc604fc9c389\": rpc error: code = NotFound desc = could not find container \"d1bb54647044b3dbf94f62820e822181bea871a3347dcd648aa7fc604fc9c389\": container with ID starting with d1bb54647044b3dbf94f62820e822181bea871a3347dcd648aa7fc604fc9c389 not found: ID does not exist"
Oct 31 23:43:58.485795 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:58.485776    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3d08af8f-75d47f56bc-w6474"]
Oct 31 23:43:58.489891 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:58.489870    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3d08af8f-75d47f56bc-w6474"]
Oct 31 23:43:58.499448 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:58.499423    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3d08af8f-5758bd6c66-2h767"]
Oct 31 23:43:58.503591 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:58.503574    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3d08af8f-5758bd6c66-2h767"]
Oct 31 23:43:59.401527 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:59.401495    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a43534d-982b-461b-a1bb-520cc0ca59ff" path="/var/lib/kubelet/pods/1a43534d-982b-461b-a1bb-520cc0ca59ff/volumes"
Oct 31 23:43:59.401953 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:43:59.401875    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e52d0ad8-6738-46a3-a64f-9824375d79f3" path="/var/lib/kubelet/pods/e52d0ad8-6738-46a3-a64f-9824375d79f3/volumes"
Oct 31 23:44:00.874323 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:00.874282    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"]
Oct 31 23:44:00.874820 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:00.874796    4301 cpu_manager.go:414] "containerMap: removing container" podUID="e52d0ad8-6738-46a3-a64f-9824375d79f3" containerName="payload-pre-processing"
Oct 31 23:44:00.874896 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:00.874822    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e52d0ad8-6738-46a3-a64f-9824375d79f3" containerName="payload-pre-processing"
Oct 31 23:44:00.874896 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:00.874834    4301 cpu_manager.go:414] "containerMap: removing container" podUID="1a43534d-982b-461b-a1bb-520cc0ca59ff" containerName="payload-processing"
Oct 31 23:44:00.874896 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:00.874843    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a43534d-982b-461b-a1bb-520cc0ca59ff" containerName="payload-processing"
Oct 31 23:44:00.874896 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:00.874876    4301 cpu_manager.go:414] "containerMap: removing container" podUID="45df8720-ddc1-42e1-aa96-e2eefa7d6e8b" containerName="maas-api"
Oct 31 23:44:00.874896 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:00.874884    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45df8720-ddc1-42e1-aa96-e2eefa7d6e8b" containerName="maas-api"
Oct 31 23:44:00.875101 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:00.874980    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="45df8720-ddc1-42e1-aa96-e2eefa7d6e8b" containerName="maas-api"
Oct 31 23:44:00.875101 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:00.874998    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="e52d0ad8-6738-46a3-a64f-9824375d79f3" containerName="payload-pre-processing"
Oct 31 23:44:00.875101 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:00.875007    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a43534d-982b-461b-a1bb-520cc0ca59ff" containerName="payload-processing"
Oct 31 23:44:00.879838 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:00.879816    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:00.889690 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:00.889668    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-44381a-openshift-default-dockercfg-4c8lp\""
Oct 31 23:44:00.894086 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:00.894058    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"]
Oct 31 23:44:01.033990 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:01.033944    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-workload-socket\") pod \"e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") " pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:01.034175 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:01.033997    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-istio-data\") pod \"e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") " pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:01.034175 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:01.034089    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-istio-envoy\") pod \"e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") " pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:01.034175 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:01.034151    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-workload-certs\") pod \"e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") " pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:01.034295 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:01.034230    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c54c461-0b9c-4732-bab8-529299440fbd-istio-podinfo\") pod \"e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") " pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:01.034295 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:01.034247    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c54c461-0b9c-4732-bab8-529299440fbd-istiod-ca-cert\") pod \"e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") " pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:01.034295 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:01.034268    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skn9t\" (UniqueName: \"kubernetes.io/projected/1c54c461-0b9c-4732-bab8-529299440fbd-kube-api-access-skn9t\") pod \"e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") " pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:01.034295 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:01.034284    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c54c461-0b9c-4732-bab8-529299440fbd-istio-token\") pod \"e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") " pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:01.034476 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:01.034328    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-credential-socket\") pod \"e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") " pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:01.135940 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:01.135858    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-workload-socket\") pod \"e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") " pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:01.135940 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:01.135906    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-istio-data\") pod \"e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") " pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:01.136139 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:01.135952    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-istio-envoy\") pod \"e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") " pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:01.136139 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:01.136025    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-workload-certs\") pod \"e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") " pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:01.136139 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:01.136070    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c54c461-0b9c-4732-bab8-529299440fbd-istio-podinfo\") pod \"e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") " pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:01.136300 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:01.136209    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c54c461-0b9c-4732-bab8-529299440fbd-istiod-ca-cert\") pod \"e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") " pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:01.136300 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:01.136259    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skn9t\" (UniqueName: \"kubernetes.io/projected/1c54c461-0b9c-4732-bab8-529299440fbd-kube-api-access-skn9t\") pod \"e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") " pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:01.136300 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:01.136290    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c54c461-0b9c-4732-bab8-529299440fbd-istio-token\") pod \"e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") " pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:01.136498 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:01.136301    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-workload-socket\") pod \"e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") " pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:01.136498 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:01.136317    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-credential-socket\") pod \"e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") " pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:01.136498 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:01.136347    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-istio-data\") pod \"e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") " pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:01.136498 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:01.136462    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-workload-certs\") pod \"e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") " pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:01.136779 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:01.136760    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-credential-socket\") pod \"e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") " pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:01.137074 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:01.137057    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c54c461-0b9c-4732-bab8-529299440fbd-istiod-ca-cert\") pod \"e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") " pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:01.138390 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:01.138335    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-istio-envoy\") pod \"e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") " pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:01.138859 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:01.138839    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c54c461-0b9c-4732-bab8-529299440fbd-istio-podinfo\") pod \"e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") " pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:01.151301 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:01.151266    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skn9t\" (UniqueName: \"kubernetes.io/projected/1c54c461-0b9c-4732-bab8-529299440fbd-kube-api-access-skn9t\") pod \"e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") " pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:01.151544 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:01.151522    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c54c461-0b9c-4732-bab8-529299440fbd-istio-token\") pod \"e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") " pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:01.198583 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:01.198549    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:01.485835 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:01.485735    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"]
Oct 31 23:44:01.488727 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:44:01.488698    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c54c461_0b9c_4732_bab8_529299440fbd.slice/crio-a41d1b45cae8ee293ca1273ca4e062ba2bbeb47b3103cf7f0dc430c7a5f55bae WatchSource:0}: Error finding container a41d1b45cae8ee293ca1273ca4e062ba2bbeb47b3103cf7f0dc430c7a5f55bae: Status 404 returned error can't find the container with id a41d1b45cae8ee293ca1273ca4e062ba2bbeb47b3103cf7f0dc430c7a5f55bae
Oct 31 23:44:01.490729 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:01.490696    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:44:01.490801 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:01.490760    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:44:01.490801 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:01.490789    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:44:02.486497 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:02.486461    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg" event={"ID":"1c54c461-0b9c-4732-bab8-529299440fbd","Type":"ContainerStarted","Data":"e1213c12f0a17135f301451c2cf949b52f41b30da7fb159488cb39e2664568e6"}
Oct 31 23:44:02.486883 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:02.486540    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg" event={"ID":"1c54c461-0b9c-4732-bab8-529299440fbd","Type":"ContainerStarted","Data":"a41d1b45cae8ee293ca1273ca4e062ba2bbeb47b3103cf7f0dc430c7a5f55bae"}
Oct 31 23:44:02.509507 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:02.509462    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg" podStartSLOduration=2.50944815 podStartE2EDuration="2.50944815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:44:02.50786608 +0000 UTC m=+3261.255055587" watchObservedRunningTime="2026-10-31 23:44:02.50944815 +0000 UTC m=+3261.256637730"
Oct 31 23:44:02.544462 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:44:02.544409    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:44:03.199584 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:03.199542    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:03.204535 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:03.204510    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:03.490421 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:03.490329    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:03.491299 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:03.491282    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:05.949608 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:05.949578    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-44381a-6df6dd4d46-vn4nq"]
Oct 31 23:44:05.953549 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:05.953529    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-44381a-6df6dd4d46-vn4nq"
Oct 31 23:44:05.960995 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:05.960732    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-44381a\""
Oct 31 23:44:05.965651 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:05.965627    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-44381a-6df6dd4d46-vn4nq"]
Oct 31 23:44:05.983241 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:05.983214    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/076b12b4-0e32-4464-9cea-7914c037f8c5-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-44381a-6df6dd4d46-vn4nq\" (UID: \"076b12b4-0e32-4464-9cea-7914c037f8c5\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-44381a-6df6dd4d46-vn4nq"
Oct 31 23:44:05.983416 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:05.983275    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtq7t\" (UniqueName: \"kubernetes.io/projected/076b12b4-0e32-4464-9cea-7914c037f8c5-kube-api-access-xtq7t\") pod \"payload-processing-e2e-authcfg-http-44381a-6df6dd4d46-vn4nq\" (UID: \"076b12b4-0e32-4464-9cea-7914c037f8c5\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-44381a-6df6dd4d46-vn4nq"
Oct 31 23:44:06.084154 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:06.084101    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/076b12b4-0e32-4464-9cea-7914c037f8c5-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-44381a-6df6dd4d46-vn4nq\" (UID: \"076b12b4-0e32-4464-9cea-7914c037f8c5\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-44381a-6df6dd4d46-vn4nq"
Oct 31 23:44:06.084350 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:06.084248    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtq7t\" (UniqueName: \"kubernetes.io/projected/076b12b4-0e32-4464-9cea-7914c037f8c5-kube-api-access-xtq7t\") pod \"payload-processing-e2e-authcfg-http-44381a-6df6dd4d46-vn4nq\" (UID: \"076b12b4-0e32-4464-9cea-7914c037f8c5\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-44381a-6df6dd4d46-vn4nq"
Oct 31 23:44:06.084985 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:06.084962    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/076b12b4-0e32-4464-9cea-7914c037f8c5-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-44381a-6df6dd4d46-vn4nq\" (UID: \"076b12b4-0e32-4464-9cea-7914c037f8c5\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-44381a-6df6dd4d46-vn4nq"
Oct 31 23:44:06.097321 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:06.097285    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtq7t\" (UniqueName: \"kubernetes.io/projected/076b12b4-0e32-4464-9cea-7914c037f8c5-kube-api-access-xtq7t\") pod \"payload-processing-e2e-authcfg-http-44381a-6df6dd4d46-vn4nq\" (UID: \"076b12b4-0e32-4464-9cea-7914c037f8c5\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-44381a-6df6dd4d46-vn4nq"
Oct 31 23:44:06.268962 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:06.268864    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-44381a-6df6dd4d46-vn4nq"
Oct 31 23:44:06.558684 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:06.558659    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-44381a-6df6dd4d46-vn4nq"]
Oct 31 23:44:06.561402 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:44:06.561370    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod076b12b4_0e32_4464_9cea_7914c037f8c5.slice/crio-d451cee7521bf6ec9c01481791d95949f950e02fac483b4420a5c15692065996 WatchSource:0}: Error finding container d451cee7521bf6ec9c01481791d95949f950e02fac483b4420a5c15692065996: Status 404 returned error can't find the container with id d451cee7521bf6ec9c01481791d95949f950e02fac483b4420a5c15692065996
Oct 31 23:44:07.507651 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:07.507618    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-44381a-6df6dd4d46-vn4nq" event={"ID":"076b12b4-0e32-4464-9cea-7914c037f8c5","Type":"ContainerStarted","Data":"50a8b174184fd35258f0192dd7c365313a874015ee655c7665364cb8034a0776"}
Oct 31 23:44:07.508116 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:07.507672    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-44381a-6df6dd4d46-vn4nq" event={"ID":"076b12b4-0e32-4464-9cea-7914c037f8c5","Type":"ContainerStarted","Data":"d451cee7521bf6ec9c01481791d95949f950e02fac483b4420a5c15692065996"}
Oct 31 23:44:07.508116 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:07.507693    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-44381a-6df6dd4d46-vn4nq"
Oct 31 23:44:07.524620 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:07.524572    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-44381a-6df6dd4d46-vn4nq" podStartSLOduration=2.524556282 podStartE2EDuration="2.524556282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:44:07.523527477 +0000 UTC m=+3266.270716981" watchObservedRunningTime="2026-10-31 23:44:07.524556282 +0000 UTC m=+3266.271745787"
Oct 31 23:44:07.913393 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:07.912790    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:44:17.573770 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:17.573732    4301 generic.go:356] "Generic (PLEG): container finished" podID="1c54c461-0b9c-4732-bab8-529299440fbd" containerID="e1213c12f0a17135f301451c2cf949b52f41b30da7fb159488cb39e2664568e6" exitCode=0
Oct 31 23:44:17.574321 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:17.573793    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg" event={"ID":"1c54c461-0b9c-4732-bab8-529299440fbd","Type":"ContainerDied","Data":"e1213c12f0a17135f301451c2cf949b52f41b30da7fb159488cb39e2664568e6"}
Oct 31 23:44:17.574321 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:17.574242    4301 scope.go:122] "RemoveContainer" containerID="e1213c12f0a17135f301451c2cf949b52f41b30da7fb159488cb39e2664568e6"
Oct 31 23:44:17.575430 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:17.575400    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:44:17.575537 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:17.575487    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:44:17.575610 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:17.575532    4301 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 23:44:18.491085 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:18.491056    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:18.514752 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:18.514730    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-44381a-6df6dd4d46-vn4nq"
Oct 31 23:44:18.580940 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:18.580904    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg" event={"ID":"1c54c461-0b9c-4732-bab8-529299440fbd","Type":"ContainerStarted","Data":"f301acf959915218874a5904710b137acb355f102b5cbb826ad9e326ff221feb"}
Oct 31 23:44:19.199295 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:19.199258    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:19.200527 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:19.200499    4301 patch_prober.go:29] interesting pod/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 31 23:44:19.200653 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:19.200557    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg" podUID="1c54c461-0b9c-4732-bab8-529299440fbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 31 23:44:20.199063 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:20.199025    4301 patch_prober.go:29] interesting pod/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 31 23:44:20.199653 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:20.199085    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg" podUID="1c54c461-0b9c-4732-bab8-529299440fbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 31 23:44:21.199711 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:21.199671    4301 patch_prober.go:29] interesting pod/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 31 23:44:21.200161 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:21.199725    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg" podUID="1c54c461-0b9c-4732-bab8-529299440fbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 31 23:44:22.199468 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:22.199432    4301 patch_prober.go:29] interesting pod/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 31 23:44:22.199666 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:22.199483    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg" podUID="1c54c461-0b9c-4732-bab8-529299440fbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 31 23:44:23.199353 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:23.199310    4301 patch_prober.go:29] interesting pod/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 31 23:44:23.199846 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:23.199395    4301 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg" podUID="1c54c461-0b9c-4732-bab8-529299440fbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 31 23:44:24.204016 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:24.203933    4301 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:24.204530 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:24.204219    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:24.205094 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:24.205079    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:44:32.544772 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:44:32.544724    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:44:41.902600 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:41.902570    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s9nl_1f0042ee-6258-4aa8-af48-7568f2f2e8a1/ovn-acl-logging/0.log"
Oct 31 23:44:41.906462 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:44:41.906432    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7s9nl_1f0042ee-6258-4aa8-af48-7568f2f2e8a1/ovn-acl-logging/0.log"
Oct 31 23:45:00.145376 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:00.145311    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891505-bxcqz"]
Oct 31 23:45:00.148896 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:00.148873    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891505-bxcqz"
Oct 31 23:45:00.151553 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:00.151527    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9nfr7\""
Oct 31 23:45:00.157505 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:00.157471    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891505-bxcqz"]
Oct 31 23:45:00.238257 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:00.238218    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnfzx\" (UniqueName: \"kubernetes.io/projected/01a2e1b1-ba9d-4215-9517-12bdacb8ac27-kube-api-access-fnfzx\") pod \"maas-api-key-cleanup-29891505-bxcqz\" (UID: \"01a2e1b1-ba9d-4215-9517-12bdacb8ac27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891505-bxcqz"
Oct 31 23:45:00.238476 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:00.238275    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a2e1b1-ba9d-4215-9517-12bdacb8ac27-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891505-bxcqz\" (UID: \"01a2e1b1-ba9d-4215-9517-12bdacb8ac27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891505-bxcqz"
Oct 31 23:45:00.338938 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:00.338900    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnfzx\" (UniqueName: \"kubernetes.io/projected/01a2e1b1-ba9d-4215-9517-12bdacb8ac27-kube-api-access-fnfzx\") pod \"maas-api-key-cleanup-29891505-bxcqz\" (UID: \"01a2e1b1-ba9d-4215-9517-12bdacb8ac27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891505-bxcqz"
Oct 31 23:45:00.339120 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:00.338960    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a2e1b1-ba9d-4215-9517-12bdacb8ac27-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891505-bxcqz\" (UID: \"01a2e1b1-ba9d-4215-9517-12bdacb8ac27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891505-bxcqz"
Oct 31 23:45:00.339563 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:00.339545    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a2e1b1-ba9d-4215-9517-12bdacb8ac27-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891505-bxcqz\" (UID: \"01a2e1b1-ba9d-4215-9517-12bdacb8ac27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891505-bxcqz"
Oct 31 23:45:00.350415 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:00.350393    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnfzx\" (UniqueName: \"kubernetes.io/projected/01a2e1b1-ba9d-4215-9517-12bdacb8ac27-kube-api-access-fnfzx\") pod \"maas-api-key-cleanup-29891505-bxcqz\" (UID: \"01a2e1b1-ba9d-4215-9517-12bdacb8ac27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891505-bxcqz"
Oct 31 23:45:00.467100 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:00.467012    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891505-bxcqz"
Oct 31 23:45:00.740434 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:00.740314    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891505-bxcqz"]
Oct 31 23:45:00.744021 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:45:00.743990    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a2e1b1_ba9d_4215_9517_12bdacb8ac27.slice/crio-94437a64f2ca9849fb76013fa12fba105d28958a80af949355e85fd77ff6d7f5 WatchSource:0}: Error finding container 94437a64f2ca9849fb76013fa12fba105d28958a80af949355e85fd77ff6d7f5: Status 404 returned error can't find the container with id 94437a64f2ca9849fb76013fa12fba105d28958a80af949355e85fd77ff6d7f5
Oct 31 23:45:00.761238 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:00.761204    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891505-bxcqz" event={"ID":"01a2e1b1-ba9d-4215-9517-12bdacb8ac27","Type":"ContainerStarted","Data":"94437a64f2ca9849fb76013fa12fba105d28958a80af949355e85fd77ff6d7f5"}
Oct 31 23:45:01.766823 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:01.766791    4301 generic.go:356] "Generic (PLEG): container finished" podID="01a2e1b1-ba9d-4215-9517-12bdacb8ac27" containerID="d5705850c7c166931ba7732c79b47d428517374624b30b7a8c457d8e556e96e5" exitCode=60
Oct 31 23:45:01.767326 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:01.766878    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891505-bxcqz" event={"ID":"01a2e1b1-ba9d-4215-9517-12bdacb8ac27","Type":"ContainerDied","Data":"d5705850c7c166931ba7732c79b47d428517374624b30b7a8c457d8e556e96e5"}
Oct 31 23:45:01.767326 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:01.767135    4301 scope.go:122] "RemoveContainer" containerID="d5705850c7c166931ba7732c79b47d428517374624b30b7a8c457d8e556e96e5"
Oct 31 23:45:02.545160 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:45:02.545127    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:45:02.773516 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:02.773485    4301 generic.go:356] "Generic (PLEG): container finished" podID="01a2e1b1-ba9d-4215-9517-12bdacb8ac27" containerID="986068e17c4ddf590e9342c7d8958353739bcab8aaa05147920b33815c4c27e8" exitCode=60
Oct 31 23:45:02.773970 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:02.773530    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891505-bxcqz" event={"ID":"01a2e1b1-ba9d-4215-9517-12bdacb8ac27","Type":"ContainerDied","Data":"986068e17c4ddf590e9342c7d8958353739bcab8aaa05147920b33815c4c27e8"}
Oct 31 23:45:02.773970 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:02.773586    4301 scope.go:122] "RemoveContainer" containerID="d5705850c7c166931ba7732c79b47d428517374624b30b7a8c457d8e556e96e5"
Oct 31 23:45:02.773970 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:02.773819    4301 scope.go:122] "RemoveContainer" containerID="986068e17c4ddf590e9342c7d8958353739bcab8aaa05147920b33815c4c27e8"
Oct 31 23:45:02.774095 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:45:02.774039    4301 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891505-bxcqz_odh-ai-gateway-infra(01a2e1b1-ba9d-4215-9517-12bdacb8ac27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891505-bxcqz" podUID="01a2e1b1-ba9d-4215-9517-12bdacb8ac27"
Oct 31 23:45:03.779088 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:03.779063    4301 scope.go:122] "RemoveContainer" containerID="986068e17c4ddf590e9342c7d8958353739bcab8aaa05147920b33815c4c27e8"
Oct 31 23:45:03.779486 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:45:03.779272    4301 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891505-bxcqz_odh-ai-gateway-infra(01a2e1b1-ba9d-4215-9517-12bdacb8ac27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891505-bxcqz" podUID="01a2e1b1-ba9d-4215-9517-12bdacb8ac27"
Oct 31 23:45:17.397812 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:17.397773    4301 scope.go:122] "RemoveContainer" containerID="986068e17c4ddf590e9342c7d8958353739bcab8aaa05147920b33815c4c27e8"
Oct 31 23:45:17.845414 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:17.845318    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891505-bxcqz" event={"ID":"01a2e1b1-ba9d-4215-9517-12bdacb8ac27","Type":"ContainerStarted","Data":"041dd2ed3870aa914c242b03af766d75880a5f0be8f81172a71976a09950bba3"}
Oct 31 23:45:17.865066 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:17.865018    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891505-bxcqz" podStartSLOduration=17.865004922 podStartE2EDuration="17.865004922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:45:17.863030585 +0000 UTC m=+3336.610220089" watchObservedRunningTime="2026-10-31 23:45:17.865004922 +0000 UTC m=+3336.612194428"
Oct 31 23:45:18.851095 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:18.851061    4301 generic.go:356] "Generic (PLEG): container finished" podID="01a2e1b1-ba9d-4215-9517-12bdacb8ac27" containerID="041dd2ed3870aa914c242b03af766d75880a5f0be8f81172a71976a09950bba3" exitCode=60
Oct 31 23:45:18.851612 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:18.851139    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891505-bxcqz" event={"ID":"01a2e1b1-ba9d-4215-9517-12bdacb8ac27","Type":"ContainerDied","Data":"041dd2ed3870aa914c242b03af766d75880a5f0be8f81172a71976a09950bba3"}
Oct 31 23:45:18.851612 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:18.851211    4301 scope.go:122] "RemoveContainer" containerID="986068e17c4ddf590e9342c7d8958353739bcab8aaa05147920b33815c4c27e8"
Oct 31 23:45:18.851612 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:18.851564    4301 scope.go:122] "RemoveContainer" containerID="041dd2ed3870aa914c242b03af766d75880a5f0be8f81172a71976a09950bba3"
Oct 31 23:45:18.851797 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:45:18.851774    4301 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891505-bxcqz_odh-ai-gateway-infra(01a2e1b1-ba9d-4215-9517-12bdacb8ac27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891505-bxcqz" podUID="01a2e1b1-ba9d-4215-9517-12bdacb8ac27"
Oct 31 23:45:18.879527 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:18.879499    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891505-bxcqz"]
Oct 31 23:45:19.975393 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:19.975347    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891505-bxcqz"
Oct 31 23:45:20.125534 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:20.125506    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01a2e1b1-ba9d-4215-9517-12bdacb8ac27-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a2e1b1-ba9d-4215-9517-12bdacb8ac27-maas-api-service-ca\") pod \"01a2e1b1-ba9d-4215-9517-12bdacb8ac27\" (UID: \"01a2e1b1-ba9d-4215-9517-12bdacb8ac27\") "
Oct 31 23:45:20.125739 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:20.125589    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a2e1b1-ba9d-4215-9517-12bdacb8ac27-kube-api-access-fnfzx\" (UniqueName: \"kubernetes.io/projected/01a2e1b1-ba9d-4215-9517-12bdacb8ac27-kube-api-access-fnfzx\") pod \"01a2e1b1-ba9d-4215-9517-12bdacb8ac27\" (UID: \"01a2e1b1-ba9d-4215-9517-12bdacb8ac27\") "
Oct 31 23:45:20.125966 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:20.125935    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a2e1b1-ba9d-4215-9517-12bdacb8ac27-maas-api-service-ca" pod "01a2e1b1-ba9d-4215-9517-12bdacb8ac27" (UID: "01a2e1b1-ba9d-4215-9517-12bdacb8ac27"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:45:20.127705 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:20.127684    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a2e1b1-ba9d-4215-9517-12bdacb8ac27-kube-api-access-fnfzx" pod "01a2e1b1-ba9d-4215-9517-12bdacb8ac27" (UID: "01a2e1b1-ba9d-4215-9517-12bdacb8ac27"). InnerVolumeSpecName "kube-api-access-fnfzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:45:20.226249 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:20.226215    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnfzx\" (UniqueName: \"kubernetes.io/projected/01a2e1b1-ba9d-4215-9517-12bdacb8ac27-kube-api-access-fnfzx\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:45:20.226249 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:20.226244    4301 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a2e1b1-ba9d-4215-9517-12bdacb8ac27-maas-api-service-ca\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:45:20.861997 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:20.861967    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891505-bxcqz"
Oct 31 23:45:20.862207 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:20.861967    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891505-bxcqz" event={"ID":"01a2e1b1-ba9d-4215-9517-12bdacb8ac27","Type":"ContainerDied","Data":"94437a64f2ca9849fb76013fa12fba105d28958a80af949355e85fd77ff6d7f5"}
Oct 31 23:45:20.862207 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:20.862101    4301 scope.go:122] "RemoveContainer" containerID="041dd2ed3870aa914c242b03af766d75880a5f0be8f81172a71976a09950bba3"
Oct 31 23:45:20.881737 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:20.881711    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891505-bxcqz"]
Oct 31 23:45:20.887380 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:20.887346    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891505-bxcqz"]
Oct 31 23:45:21.401884 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:21.401856    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01a2e1b1-ba9d-4215-9517-12bdacb8ac27" path="/var/lib/kubelet/pods/01a2e1b1-ba9d-4215-9517-12bdacb8ac27/volumes"
Oct 31 23:45:32.545302 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:45:32.545264    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:45:49.961617 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:49.961578    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-44381a-6df6dd4d46-vn4nq"]
Oct 31 23:45:54.962131 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:54.962046    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-44381a-6df6dd4d46-vn4nq" podUID="076b12b4-0e32-4464-9cea-7914c037f8c5" containerName="payload-processing" containerID="cri-o://50a8b174184fd35258f0192dd7c365313a874015ee655c7665364cb8034a0776" gracePeriod=25
Oct 31 23:45:55.217013 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:55.216922    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-44381a-6df6dd4d46-vn4nq"
Oct 31 23:45:55.344953 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:55.344924    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/076b12b4-0e32-4464-9cea-7914c037f8c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/076b12b4-0e32-4464-9cea-7914c037f8c5-plugins-config-volume\") pod \"076b12b4-0e32-4464-9cea-7914c037f8c5\" (UID: \"076b12b4-0e32-4464-9cea-7914c037f8c5\") "
Oct 31 23:45:55.345125 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:55.344987    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/076b12b4-0e32-4464-9cea-7914c037f8c5-kube-api-access-xtq7t\" (UniqueName: \"kubernetes.io/projected/076b12b4-0e32-4464-9cea-7914c037f8c5-kube-api-access-xtq7t\") pod \"076b12b4-0e32-4464-9cea-7914c037f8c5\" (UID: \"076b12b4-0e32-4464-9cea-7914c037f8c5\") "
Oct 31 23:45:55.345290 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:55.345268    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/076b12b4-0e32-4464-9cea-7914c037f8c5-plugins-config-volume" pod "076b12b4-0e32-4464-9cea-7914c037f8c5" (UID: "076b12b4-0e32-4464-9cea-7914c037f8c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:45:55.345390 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:55.345353    4301 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/076b12b4-0e32-4464-9cea-7914c037f8c5-plugins-config-volume\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:45:55.347100 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:55.347070    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/076b12b4-0e32-4464-9cea-7914c037f8c5-kube-api-access-xtq7t" pod "076b12b4-0e32-4464-9cea-7914c037f8c5" (UID: "076b12b4-0e32-4464-9cea-7914c037f8c5"). InnerVolumeSpecName "kube-api-access-xtq7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:45:55.446040 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:55.446012    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtq7t\" (UniqueName: \"kubernetes.io/projected/076b12b4-0e32-4464-9cea-7914c037f8c5-kube-api-access-xtq7t\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:45:56.009771 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:56.009732    4301 generic.go:356] "Generic (PLEG): container finished" podID="076b12b4-0e32-4464-9cea-7914c037f8c5" containerID="50a8b174184fd35258f0192dd7c365313a874015ee655c7665364cb8034a0776" exitCode=0
Oct 31 23:45:56.010284 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:56.009780    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-44381a-6df6dd4d46-vn4nq"
Oct 31 23:45:56.010284 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:56.009805    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-44381a-6df6dd4d46-vn4nq" event={"ID":"076b12b4-0e32-4464-9cea-7914c037f8c5","Type":"ContainerDied","Data":"50a8b174184fd35258f0192dd7c365313a874015ee655c7665364cb8034a0776"}
Oct 31 23:45:56.010284 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:56.009856    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-44381a-6df6dd4d46-vn4nq" event={"ID":"076b12b4-0e32-4464-9cea-7914c037f8c5","Type":"ContainerDied","Data":"d451cee7521bf6ec9c01481791d95949f950e02fac483b4420a5c15692065996"}
Oct 31 23:45:56.010284 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:56.009887    4301 scope.go:122] "RemoveContainer" containerID="50a8b174184fd35258f0192dd7c365313a874015ee655c7665364cb8034a0776"
Oct 31 23:45:56.018151 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:56.018133    4301 scope.go:122] "RemoveContainer" containerID="50a8b174184fd35258f0192dd7c365313a874015ee655c7665364cb8034a0776"
Oct 31 23:45:56.018419 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:45:56.018400    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50a8b174184fd35258f0192dd7c365313a874015ee655c7665364cb8034a0776\": container with ID starting with 50a8b174184fd35258f0192dd7c365313a874015ee655c7665364cb8034a0776 not found: ID does not exist" containerID="50a8b174184fd35258f0192dd7c365313a874015ee655c7665364cb8034a0776"
Oct 31 23:45:56.018479 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:56.018428    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50a8b174184fd35258f0192dd7c365313a874015ee655c7665364cb8034a0776"} err="failed to get container status \"50a8b174184fd35258f0192dd7c365313a874015ee655c7665364cb8034a0776\": rpc error: code = NotFound desc = could not find container \"50a8b174184fd35258f0192dd7c365313a874015ee655c7665364cb8034a0776\": container with ID starting with 50a8b174184fd35258f0192dd7c365313a874015ee655c7665364cb8034a0776 not found: ID does not exist"
Oct 31 23:45:56.024519 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:56.024498    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-44381a-6df6dd4d46-vn4nq"]
Oct 31 23:45:56.029339 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:56.029318    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-44381a-6df6dd4d46-vn4nq"]
Oct 31 23:45:56.680582 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:56.680550    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:45:57.402084 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:45:57.402051    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="076b12b4-0e32-4464-9cea-7914c037f8c5" path="/var/lib/kubelet/pods/076b12b4-0e32-4464-9cea-7914c037f8c5/volumes"
Oct 31 23:46:02.318288 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:02.318255    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"]
Oct 31 23:46:02.318810 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:02.318528    4301 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg" podUID="1c54c461-0b9c-4732-bab8-529299440fbd" containerName="istio-proxy" containerID="cri-o://f301acf959915218874a5904710b137acb355f102b5cbb826ad9e326ff221feb" gracePeriod=30
Oct 31 23:46:02.545214 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:46:02.545177    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:46:04.417448 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:04.417415    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:46:07.565957 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:07.565934    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:46:07.602416 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:07.602339    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c54c461-0b9c-4732-bab8-529299440fbd-istio-token\" (UniqueName: \"kubernetes.io/projected/1c54c461-0b9c-4732-bab8-529299440fbd-istio-token\") pod \"1c54c461-0b9c-4732-bab8-529299440fbd\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") "
Oct 31 23:46:07.602416 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:07.602398    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c54c461-0b9c-4732-bab8-529299440fbd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c54c461-0b9c-4732-bab8-529299440fbd-istiod-ca-cert\") pod \"1c54c461-0b9c-4732-bab8-529299440fbd\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") "
Oct 31 23:46:07.602630 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:07.602427    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-credential-socket\") pod \"1c54c461-0b9c-4732-bab8-529299440fbd\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") "
Oct 31 23:46:07.602630 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:07.602473    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-istio-data\") pod \"1c54c461-0b9c-4732-bab8-529299440fbd\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") "
Oct 31 23:46:07.602630 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:07.602504    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-istio-envoy\") pod \"1c54c461-0b9c-4732-bab8-529299440fbd\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") "
Oct 31 23:46:07.602630 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:07.602533    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c54c461-0b9c-4732-bab8-529299440fbd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c54c461-0b9c-4732-bab8-529299440fbd-istio-podinfo\") pod \"1c54c461-0b9c-4732-bab8-529299440fbd\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") "
Oct 31 23:46:07.602630 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:07.602571    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-workload-socket\") pod \"1c54c461-0b9c-4732-bab8-529299440fbd\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") "
Oct 31 23:46:07.602630 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:07.602604    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c54c461-0b9c-4732-bab8-529299440fbd-kube-api-access-skn9t\" (UniqueName: \"kubernetes.io/projected/1c54c461-0b9c-4732-bab8-529299440fbd-kube-api-access-skn9t\") pod \"1c54c461-0b9c-4732-bab8-529299440fbd\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") "
Oct 31 23:46:07.602957 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:07.602647    4301 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-workload-certs\") pod \"1c54c461-0b9c-4732-bab8-529299440fbd\" (UID: \"1c54c461-0b9c-4732-bab8-529299440fbd\") "
Oct 31 23:46:07.602957 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:07.602766    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-credential-socket" pod "1c54c461-0b9c-4732-bab8-529299440fbd" (UID: "1c54c461-0b9c-4732-bab8-529299440fbd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:46:07.602957 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:07.602792    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-istio-data" pod "1c54c461-0b9c-4732-bab8-529299440fbd" (UID: "1c54c461-0b9c-4732-bab8-529299440fbd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:46:07.602957 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:07.602907    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-workload-socket" pod "1c54c461-0b9c-4732-bab8-529299440fbd" (UID: "1c54c461-0b9c-4732-bab8-529299440fbd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:46:07.602957 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:07.602834    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c54c461-0b9c-4732-bab8-529299440fbd-istiod-ca-cert" pod "1c54c461-0b9c-4732-bab8-529299440fbd" (UID: "1c54c461-0b9c-4732-bab8-529299440fbd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 23:46:07.603242 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:07.603007    4301 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c54c461-0b9c-4732-bab8-529299440fbd-istiod-ca-cert\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:46:07.603242 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:07.603025    4301 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-credential-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:46:07.603242 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:07.603038    4301 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-istio-data\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:46:07.603242 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:07.603062    4301 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-workload-socket\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:46:07.603242 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:07.603106    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-workload-certs" pod "1c54c461-0b9c-4732-bab8-529299440fbd" (UID: "1c54c461-0b9c-4732-bab8-529299440fbd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:46:07.604845 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:07.604819    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c54c461-0b9c-4732-bab8-529299440fbd-kube-api-access-skn9t" pod "1c54c461-0b9c-4732-bab8-529299440fbd" (UID: "1c54c461-0b9c-4732-bab8-529299440fbd"). InnerVolumeSpecName "kube-api-access-skn9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:46:07.604968 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:07.604907    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c54c461-0b9c-4732-bab8-529299440fbd-istio-token" pod "1c54c461-0b9c-4732-bab8-529299440fbd" (UID: "1c54c461-0b9c-4732-bab8-529299440fbd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 23:46:07.605107 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:07.605092    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c54c461-0b9c-4732-bab8-529299440fbd-istio-podinfo" pod "1c54c461-0b9c-4732-bab8-529299440fbd" (UID: "1c54c461-0b9c-4732-bab8-529299440fbd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 23:46:07.605176 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:07.605099    4301 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-istio-envoy" pod "1c54c461-0b9c-4732-bab8-529299440fbd" (UID: "1c54c461-0b9c-4732-bab8-529299440fbd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 23:46:07.703747 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:07.703711    4301 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-workload-certs\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:46:07.703747 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:07.703742    4301 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c54c461-0b9c-4732-bab8-529299440fbd-istio-token\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:46:07.703747 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:07.703751    4301 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c54c461-0b9c-4732-bab8-529299440fbd-istio-envoy\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:46:07.703975 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:07.703760    4301 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c54c461-0b9c-4732-bab8-529299440fbd-istio-podinfo\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:46:07.703975 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:07.703769    4301 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skn9t\" (UniqueName: \"kubernetes.io/projected/1c54c461-0b9c-4732-bab8-529299440fbd-kube-api-access-skn9t\") on node \"ip-10-0-135-163.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 23:46:08.095236 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:08.095201    4301 generic.go:356] "Generic (PLEG): container finished" podID="1c54c461-0b9c-4732-bab8-529299440fbd" containerID="f301acf959915218874a5904710b137acb355f102b5cbb826ad9e326ff221feb" exitCode=0
Oct 31 23:46:08.095469 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:08.095252    4301 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"
Oct 31 23:46:08.095469 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:08.095280    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg" event={"ID":"1c54c461-0b9c-4732-bab8-529299440fbd","Type":"ContainerDied","Data":"f301acf959915218874a5904710b137acb355f102b5cbb826ad9e326ff221feb"}
Oct 31 23:46:08.095469 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:08.095331    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg" event={"ID":"1c54c461-0b9c-4732-bab8-529299440fbd","Type":"ContainerDied","Data":"a41d1b45cae8ee293ca1273ca4e062ba2bbeb47b3103cf7f0dc430c7a5f55bae"}
Oct 31 23:46:08.095469 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:08.095352    4301 scope.go:122] "RemoveContainer" containerID="f301acf959915218874a5904710b137acb355f102b5cbb826ad9e326ff221feb"
Oct 31 23:46:08.105027 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:08.104991    4301 scope.go:122] "RemoveContainer" containerID="e1213c12f0a17135f301451c2cf949b52f41b30da7fb159488cb39e2664568e6"
Oct 31 23:46:08.113177 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:08.113163    4301 scope.go:122] "RemoveContainer" containerID="f301acf959915218874a5904710b137acb355f102b5cbb826ad9e326ff221feb"
Oct 31 23:46:08.113488 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:46:08.113466    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f301acf959915218874a5904710b137acb355f102b5cbb826ad9e326ff221feb\": container with ID starting with f301acf959915218874a5904710b137acb355f102b5cbb826ad9e326ff221feb not found: ID does not exist" containerID="f301acf959915218874a5904710b137acb355f102b5cbb826ad9e326ff221feb"
Oct 31 23:46:08.113575 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:08.113494    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f301acf959915218874a5904710b137acb355f102b5cbb826ad9e326ff221feb"} err="failed to get container status \"f301acf959915218874a5904710b137acb355f102b5cbb826ad9e326ff221feb\": rpc error: code = NotFound desc = could not find container \"f301acf959915218874a5904710b137acb355f102b5cbb826ad9e326ff221feb\": container with ID starting with f301acf959915218874a5904710b137acb355f102b5cbb826ad9e326ff221feb not found: ID does not exist"
Oct 31 23:46:08.113575 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:08.113511    4301 scope.go:122] "RemoveContainer" containerID="e1213c12f0a17135f301451c2cf949b52f41b30da7fb159488cb39e2664568e6"
Oct 31 23:46:08.113784 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:46:08.113766    4301 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1213c12f0a17135f301451c2cf949b52f41b30da7fb159488cb39e2664568e6\": container with ID starting with e1213c12f0a17135f301451c2cf949b52f41b30da7fb159488cb39e2664568e6 not found: ID does not exist" containerID="e1213c12f0a17135f301451c2cf949b52f41b30da7fb159488cb39e2664568e6"
Oct 31 23:46:08.113842 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:08.113793    4301 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1213c12f0a17135f301451c2cf949b52f41b30da7fb159488cb39e2664568e6"} err="failed to get container status \"e1213c12f0a17135f301451c2cf949b52f41b30da7fb159488cb39e2664568e6\": rpc error: code = NotFound desc = could not find container \"e1213c12f0a17135f301451c2cf949b52f41b30da7fb159488cb39e2664568e6\": container with ID starting with e1213c12f0a17135f301451c2cf949b52f41b30da7fb159488cb39e2664568e6 not found: ID does not exist"
Oct 31 23:46:08.117581 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:08.117561    4301 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"]
Oct 31 23:46:08.121474 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:08.121454    4301 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-44381a-openshift-default-54f59497b9-vmrxg"]
Oct 31 23:46:09.402090 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:09.402058    4301 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c54c461-0b9c-4732-bab8-529299440fbd" path="/var/lib/kubelet/pods/1c54c461-0b9c-4732-bab8-529299440fbd/volumes"
Oct 31 23:46:09.911272 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:09.911244    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:46:25.481766 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:25.481729    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:46:29.544537 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:29.544494    4301 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-s6nvl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 23:46:29.548500 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:29.548456    4301 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-s6nvl" podUID="99873d5d-87c2-423a-9a44-1b3fc83d0b46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 23:46:32.545433 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:46:32.545398    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:46:52.581767 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:46:52.581726    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-64lft"]
Oct 31 23:47:02.544648 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:47:02.544609    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:47:03.998182 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:47:03.998150    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-bjjv6_77ef2de4-0420-42ac-8b0f-eb7a117b7c4c/manager/0.log"
Oct 31 23:47:04.638065 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:47:04.638037    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-nrdhv_d2e1fa43-f1e5-4da5-a89e-cb3ab82eb88e/manager/0.log"
Oct 31 23:47:05.832063 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:47:05.832031    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6bf6d56f9f-9zkqk_24ca21f3-521c-4bf8-80b4-1c0a7757a0bd/maas-api/0.log"
Oct 31 23:47:09.416752 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:47:09.416725    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-lh6zd_85434a39-a2bd-43e2-980e-2983c549f656/kuadrant-console-plugin/0.log"
Oct 31 23:47:09.853153 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:47:09.853069    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-64lft_800e23f4-edde-455c-bd92-acb8ad2cc8af/limitador/0.log"
Oct 31 23:47:11.773921 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:47:11.773890    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-689d5458bb-hzp7b_7230a094-2ba4-41af-8bec-2d8b2b266ca9/kube-auth-proxy/0.log"
Oct 31 23:47:12.147955 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:47:12.147923    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-6qscl_e6602036-1c07-4882-ab1a-1dbbd0a857f5/istio-proxy/1.log"
Oct 31 23:47:13.188557 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:47:13.188528    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-s6nvl_99873d5d-87c2-423a-9a44-1b3fc83d0b46/istio-proxy/0.log"
Oct 31 23:47:13.751403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:47:13.751373    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-z9tfc_8ea56eff-0929-4f33-9adc-f10042fb1b96/istio-proxy/1.log"
Oct 31 23:47:15.224839 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:47:15.224807    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-6qscl_e6602036-1c07-4882-ab1a-1dbbd0a857f5/istio-proxy/0.log"
Oct 31 23:47:16.145230 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:47:16.145202    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-z9tfc_8ea56eff-0929-4f33-9adc-f10042fb1b96/istio-proxy/0.log"
Oct 31 23:47:28.893200 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:47:28.893112    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-bjjv6_77ef2de4-0420-42ac-8b0f-eb7a117b7c4c/manager/0.log"
Oct 31 23:47:29.495458 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:47:29.495425    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-nrdhv_d2e1fa43-f1e5-4da5-a89e-cb3ab82eb88e/manager/0.log"
Oct 31 23:47:30.240492 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:47:30.240462    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6bf6d56f9f-9zkqk_24ca21f3-521c-4bf8-80b4-1c0a7757a0bd/maas-api/0.log"
Oct 31 23:47:32.544701 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:47:32.544668    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:47:33.405038 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:47:33.405007    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-lh6zd_85434a39-a2bd-43e2-980e-2983c549f656/kuadrant-console-plugin/0.log"
Oct 31 23:47:33.770494 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:47:33.770418    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-64lft_800e23f4-edde-455c-bd92-acb8ad2cc8af/limitador/0.log"
Oct 31 23:47:35.070799 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:47:35.070765    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-689d5458bb-hzp7b_7230a094-2ba4-41af-8bec-2d8b2b266ca9/kube-auth-proxy/0.log"
Oct 31 23:47:35.450418 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:47:35.450384    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-6qscl_e6602036-1c07-4882-ab1a-1dbbd0a857f5/istio-proxy/1.log"
Oct 31 23:47:36.342844 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:47:36.342812    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-s6nvl_99873d5d-87c2-423a-9a44-1b3fc83d0b46/istio-proxy/0.log"
Oct 31 23:47:36.879260 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:47:36.879227    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-z9tfc_8ea56eff-0929-4f33-9adc-f10042fb1b96/istio-proxy/1.log"
Oct 31 23:48:02.544293 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:48:02.544255    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:48:32.544831 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:48:32.544785    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:48:54.323295 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:54.323208    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nmqzm/must-gather-w8cn9"]
Oct 31 23:48:54.323786 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:54.323533    4301 cpu_manager.go:414] "containerMap: removing container" podUID="1c54c461-0b9c-4732-bab8-529299440fbd" containerName="istio-proxy"
Oct 31 23:48:54.323786 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:54.323545    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c54c461-0b9c-4732-bab8-529299440fbd" containerName="istio-proxy"
Oct 31 23:48:54.323786 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:54.323556    4301 cpu_manager.go:414] "containerMap: removing container" podUID="076b12b4-0e32-4464-9cea-7914c037f8c5" containerName="payload-processing"
Oct 31 23:48:54.323786 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:54.323562    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="076b12b4-0e32-4464-9cea-7914c037f8c5" containerName="payload-processing"
Oct 31 23:48:54.323786 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:54.323569    4301 cpu_manager.go:414] "containerMap: removing container" podUID="1c54c461-0b9c-4732-bab8-529299440fbd" containerName="istio-proxy"
Oct 31 23:48:54.323786 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:54.323573    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c54c461-0b9c-4732-bab8-529299440fbd" containerName="istio-proxy"
Oct 31 23:48:54.323786 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:54.323586    4301 cpu_manager.go:414] "containerMap: removing container" podUID="01a2e1b1-ba9d-4215-9517-12bdacb8ac27" containerName="cleanup"
Oct 31 23:48:54.323786 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:54.323591    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01a2e1b1-ba9d-4215-9517-12bdacb8ac27" containerName="cleanup"
Oct 31 23:48:54.323786 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:54.323601    4301 cpu_manager.go:414] "containerMap: removing container" podUID="01a2e1b1-ba9d-4215-9517-12bdacb8ac27" containerName="cleanup"
Oct 31 23:48:54.323786 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:54.323607    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01a2e1b1-ba9d-4215-9517-12bdacb8ac27" containerName="cleanup"
Oct 31 23:48:54.323786 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:54.323614    4301 cpu_manager.go:414] "containerMap: removing container" podUID="01a2e1b1-ba9d-4215-9517-12bdacb8ac27" containerName="cleanup"
Oct 31 23:48:54.323786 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:54.323618    4301 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01a2e1b1-ba9d-4215-9517-12bdacb8ac27" containerName="cleanup"
Oct 31 23:48:54.323786 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:54.323670    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c54c461-0b9c-4732-bab8-529299440fbd" containerName="istio-proxy"
Oct 31 23:48:54.323786 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:54.323678    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="076b12b4-0e32-4464-9cea-7914c037f8c5" containerName="payload-processing"
Oct 31 23:48:54.323786 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:54.323684    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c54c461-0b9c-4732-bab8-529299440fbd" containerName="istio-proxy"
Oct 31 23:48:54.323786 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:54.323690    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="01a2e1b1-ba9d-4215-9517-12bdacb8ac27" containerName="cleanup"
Oct 31 23:48:54.323786 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:54.323695    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="01a2e1b1-ba9d-4215-9517-12bdacb8ac27" containerName="cleanup"
Oct 31 23:48:54.323786 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:54.323701    4301 memory_manager.go:365] "RemoveStaleState removing state" podUID="01a2e1b1-ba9d-4215-9517-12bdacb8ac27" containerName="cleanup"
Oct 31 23:48:54.326702 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:54.326687    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nmqzm/must-gather-w8cn9"
Oct 31 23:48:54.329528 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:54.329510    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nmqzm\"/\"kube-root-ca.crt\""
Oct 31 23:48:54.329799 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:54.329784    4301 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nmqzm\"/\"openshift-service-ca.crt\""
Oct 31 23:48:54.329865 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:54.329821    4301 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nmqzm\"/\"default-dockercfg-sstqk\""
Oct 31 23:48:54.349407 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:54.349382    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nmqzm/must-gather-w8cn9"]
Oct 31 23:48:54.410683 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:54.410651    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gfjq\" (UniqueName: \"kubernetes.io/projected/7e981292-c9f1-49dd-89bd-a10b3da00fea-kube-api-access-5gfjq\") pod \"must-gather-w8cn9\" (UID: \"7e981292-c9f1-49dd-89bd-a10b3da00fea\") " pod="openshift-must-gather-nmqzm/must-gather-w8cn9"
Oct 31 23:48:54.410875 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:54.410769    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7e981292-c9f1-49dd-89bd-a10b3da00fea-must-gather-output\") pod \"must-gather-w8cn9\" (UID: \"7e981292-c9f1-49dd-89bd-a10b3da00fea\") " pod="openshift-must-gather-nmqzm/must-gather-w8cn9"
Oct 31 23:48:54.511791 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:54.511760    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7e981292-c9f1-49dd-89bd-a10b3da00fea-must-gather-output\") pod \"must-gather-w8cn9\" (UID: \"7e981292-c9f1-49dd-89bd-a10b3da00fea\") " pod="openshift-must-gather-nmqzm/must-gather-w8cn9"
Oct 31 23:48:54.511970 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:54.511814    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gfjq\" (UniqueName: \"kubernetes.io/projected/7e981292-c9f1-49dd-89bd-a10b3da00fea-kube-api-access-5gfjq\") pod \"must-gather-w8cn9\" (UID: \"7e981292-c9f1-49dd-89bd-a10b3da00fea\") " pod="openshift-must-gather-nmqzm/must-gather-w8cn9"
Oct 31 23:48:54.512180 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:54.512160    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7e981292-c9f1-49dd-89bd-a10b3da00fea-must-gather-output\") pod \"must-gather-w8cn9\" (UID: \"7e981292-c9f1-49dd-89bd-a10b3da00fea\") " pod="openshift-must-gather-nmqzm/must-gather-w8cn9"
Oct 31 23:48:54.523422 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:54.523399    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gfjq\" (UniqueName: \"kubernetes.io/projected/7e981292-c9f1-49dd-89bd-a10b3da00fea-kube-api-access-5gfjq\") pod \"must-gather-w8cn9\" (UID: \"7e981292-c9f1-49dd-89bd-a10b3da00fea\") " pod="openshift-must-gather-nmqzm/must-gather-w8cn9"
Oct 31 23:48:54.639816 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:54.639786    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nmqzm/must-gather-w8cn9"
Oct 31 23:48:54.915534 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:54.915459    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nmqzm/must-gather-w8cn9"]
Oct 31 23:48:54.919074 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:48:54.919046    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e981292_c9f1_49dd_89bd_a10b3da00fea.slice/crio-570de84a781494b23ef6b0e7c78c501825d30b5d8dc159d719dc1204fdea52e7 WatchSource:0}: Error finding container 570de84a781494b23ef6b0e7c78c501825d30b5d8dc159d719dc1204fdea52e7: Status 404 returned error can't find the container with id 570de84a781494b23ef6b0e7c78c501825d30b5d8dc159d719dc1204fdea52e7
Oct 31 23:48:54.921222 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:54.921207    4301 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 23:48:55.788985 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:55.788946    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nmqzm/must-gather-w8cn9" event={"ID":"7e981292-c9f1-49dd-89bd-a10b3da00fea","Type":"ContainerStarted","Data":"570de84a781494b23ef6b0e7c78c501825d30b5d8dc159d719dc1204fdea52e7"}
Oct 31 23:48:56.795671 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:56.795634    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nmqzm/must-gather-w8cn9" event={"ID":"7e981292-c9f1-49dd-89bd-a10b3da00fea","Type":"ContainerStarted","Data":"fa6e48220a85e3b498b1318e46763c19e1bf6f965a6ab7eb060ecb65530544a3"}
Oct 31 23:48:56.795671 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:56.795680    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nmqzm/must-gather-w8cn9" event={"ID":"7e981292-c9f1-49dd-89bd-a10b3da00fea","Type":"ContainerStarted","Data":"aa85efebd2479e9c3611427215ce7786b3b79d664d190e916bf581e4e17921c0"}
Oct 31 23:48:56.814124 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:56.814059    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-nmqzm/must-gather-w8cn9" podStartSLOduration=1.820749042 podStartE2EDuration="2.814040528s" totalImagesPullingTime="993.291486ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:48:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:48:56.812218263 +0000 UTC m=+3555.559407768" watchObservedRunningTime="2026-10-31 23:48:56.814040528 +0000 UTC m=+3555.561230034"
Oct 31 23:48:57.692848 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:57.692814    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qv88p_620515c1-daf9-48e9-9718-a71057b3b615/global-pull-secret-syncer/0.log"
Oct 31 23:48:57.967107 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:57.967040    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vc2j7_5f102a31-27f2-410b-9e36-2cdb8d9e852c/konnectivity-agent/0.log"
Oct 31 23:48:58.103291 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:48:58.103257    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-163.us-east-2.compute.internal_2a808dc7f3e00ce75e0f6c4e3f550fac/haproxy/0.log"
Oct 31 23:49:02.591723 ip-10-0-135-163 kubenswrapper[4301]: E1031 23:49:02.591639    4301 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mg6wn" containerName="extractor"
Oct 31 23:49:04.989997 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:04.989969    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-lh6zd_85434a39-a2bd-43e2-980e-2983c549f656/kuadrant-console-plugin/0.log"
Oct 31 23:49:05.068253 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:05.068228    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-64lft_800e23f4-edde-455c-bd92-acb8ad2cc8af/limitador/0.log"
Oct 31 23:49:07.052119 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:07.051991    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jsnjt_4f04cc51-6c46-481c-a5e3-367e11024767/node-exporter/0.log"
Oct 31 23:49:07.083020 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:07.082995    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jsnjt_4f04cc51-6c46-481c-a5e3-367e11024767/kube-rbac-proxy/0.log"
Oct 31 23:49:07.109612 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:07.109588    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jsnjt_4f04cc51-6c46-481c-a5e3-367e11024767/init-textfile/0.log"
Oct 31 23:49:07.367464 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:07.367082    4301 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nmqzm/perf-node-gather-daemonset-6228z"]
Oct 31 23:49:07.379315 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:07.379286    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-6228z"
Oct 31 23:49:07.381940 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:07.381913    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nmqzm/perf-node-gather-daemonset-6228z"]
Oct 31 23:49:07.462959 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:07.462920    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3621ed01-0229-4ebb-a6fd-a9e22cccd101-proc\") pod \"perf-node-gather-daemonset-6228z\" (UID: \"3621ed01-0229-4ebb-a6fd-a9e22cccd101\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-6228z"
Oct 31 23:49:07.463158 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:07.462969    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpjm7\" (UniqueName: \"kubernetes.io/projected/3621ed01-0229-4ebb-a6fd-a9e22cccd101-kube-api-access-qpjm7\") pod \"perf-node-gather-daemonset-6228z\" (UID: \"3621ed01-0229-4ebb-a6fd-a9e22cccd101\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-6228z"
Oct 31 23:49:07.463158 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:07.462993    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3621ed01-0229-4ebb-a6fd-a9e22cccd101-podres\") pod \"perf-node-gather-daemonset-6228z\" (UID: \"3621ed01-0229-4ebb-a6fd-a9e22cccd101\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-6228z"
Oct 31 23:49:07.463158 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:07.463038    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3621ed01-0229-4ebb-a6fd-a9e22cccd101-lib-modules\") pod \"perf-node-gather-daemonset-6228z\" (UID: \"3621ed01-0229-4ebb-a6fd-a9e22cccd101\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-6228z"
Oct 31 23:49:07.463158 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:07.463056    4301 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3621ed01-0229-4ebb-a6fd-a9e22cccd101-sys\") pod \"perf-node-gather-daemonset-6228z\" (UID: \"3621ed01-0229-4ebb-a6fd-a9e22cccd101\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-6228z"
Oct 31 23:49:07.477108 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:07.477078    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-spzc7_071935de-a5ac-4642-9f27-209332334466/prometheus-operator/0.log"
Oct 31 23:49:07.504037 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:07.504008    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-spzc7_071935de-a5ac-4642-9f27-209332334466/kube-rbac-proxy/0.log"
Oct 31 23:49:07.534194 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:07.534160    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-784qh_3625be12-bd2d-4134-b3a9-4f67638d0c4e/prometheus-operator-admission-webhook/0.log"
Oct 31 23:49:07.563910 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:07.563877    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3621ed01-0229-4ebb-a6fd-a9e22cccd101-proc\") pod \"perf-node-gather-daemonset-6228z\" (UID: \"3621ed01-0229-4ebb-a6fd-a9e22cccd101\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-6228z"
Oct 31 23:49:07.564094 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:07.563919    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpjm7\" (UniqueName: \"kubernetes.io/projected/3621ed01-0229-4ebb-a6fd-a9e22cccd101-kube-api-access-qpjm7\") pod \"perf-node-gather-daemonset-6228z\" (UID: \"3621ed01-0229-4ebb-a6fd-a9e22cccd101\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-6228z"
Oct 31 23:49:07.564094 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:07.564004    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3621ed01-0229-4ebb-a6fd-a9e22cccd101-proc\") pod \"perf-node-gather-daemonset-6228z\" (UID: \"3621ed01-0229-4ebb-a6fd-a9e22cccd101\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-6228z"
Oct 31 23:49:07.564094 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:07.564038    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3621ed01-0229-4ebb-a6fd-a9e22cccd101-podres\") pod \"perf-node-gather-daemonset-6228z\" (UID: \"3621ed01-0229-4ebb-a6fd-a9e22cccd101\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-6228z"
Oct 31 23:49:07.564222 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:07.564122    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3621ed01-0229-4ebb-a6fd-a9e22cccd101-lib-modules\") pod \"perf-node-gather-daemonset-6228z\" (UID: \"3621ed01-0229-4ebb-a6fd-a9e22cccd101\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-6228z"
Oct 31 23:49:07.564222 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:07.564137    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3621ed01-0229-4ebb-a6fd-a9e22cccd101-podres\") pod \"perf-node-gather-daemonset-6228z\" (UID: \"3621ed01-0229-4ebb-a6fd-a9e22cccd101\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-6228z"
Oct 31 23:49:07.564222 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:07.564154    4301 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3621ed01-0229-4ebb-a6fd-a9e22cccd101-sys\") pod \"perf-node-gather-daemonset-6228z\" (UID: \"3621ed01-0229-4ebb-a6fd-a9e22cccd101\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-6228z"
Oct 31 23:49:07.564222 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:07.564207    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3621ed01-0229-4ebb-a6fd-a9e22cccd101-lib-modules\") pod \"perf-node-gather-daemonset-6228z\" (UID: \"3621ed01-0229-4ebb-a6fd-a9e22cccd101\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-6228z"
Oct 31 23:49:07.564385 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:07.564250    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3621ed01-0229-4ebb-a6fd-a9e22cccd101-sys\") pod \"perf-node-gather-daemonset-6228z\" (UID: \"3621ed01-0229-4ebb-a6fd-a9e22cccd101\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-6228z"
Oct 31 23:49:07.576450 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:07.576422    4301 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpjm7\" (UniqueName: \"kubernetes.io/projected/3621ed01-0229-4ebb-a6fd-a9e22cccd101-kube-api-access-qpjm7\") pod \"perf-node-gather-daemonset-6228z\" (UID: \"3621ed01-0229-4ebb-a6fd-a9e22cccd101\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-6228z"
Oct 31 23:49:07.704778 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:07.704742    4301 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-6228z"
Oct 31 23:49:07.994873 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:07.994791    4301 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nmqzm/perf-node-gather-daemonset-6228z"]
Oct 31 23:49:07.998342 ip-10-0-135-163 kubenswrapper[4301]: W1031 23:49:07.998311    4301 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3621ed01_0229_4ebb_a6fd_a9e22cccd101.slice/crio-38a1e8ef90be96fab486ec4dc8233af776d564ef406aa22bde3f6512ce7e8912 WatchSource:0}: Error finding container 38a1e8ef90be96fab486ec4dc8233af776d564ef406aa22bde3f6512ce7e8912: Status 404 returned error can't find the container with id 38a1e8ef90be96fab486ec4dc8233af776d564ef406aa22bde3f6512ce7e8912
Oct 31 23:49:08.910754 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:08.907680    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-6228z" event={"ID":"3621ed01-0229-4ebb-a6fd-a9e22cccd101","Type":"ContainerStarted","Data":"be4e7c6c2209d8d3f1c2bd62f01f343863bacd08e4a3450c22b4c63baf3b4ba7"}
Oct 31 23:49:08.910754 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:08.907748    4301 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-6228z" event={"ID":"3621ed01-0229-4ebb-a6fd-a9e22cccd101","Type":"ContainerStarted","Data":"38a1e8ef90be96fab486ec4dc8233af776d564ef406aa22bde3f6512ce7e8912"}
Oct 31 23:49:08.910754 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:08.907789    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-6228z"
Oct 31 23:49:08.927390 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:08.926436    4301 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-6228z" podStartSLOduration=1.92642165 podStartE2EDuration="1.92642165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 23:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:49:08.925994992 +0000 UTC m=+3567.673184497" watchObservedRunningTime="2026-10-31 23:49:08.92642165 +0000 UTC m=+3567.673611163"
Oct 31 23:49:10.160784 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:10.160750    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5b7bc8f456-npf6x_f151329a-9922-4576-8de2-1f8b2034934e/console/0.log"
Oct 31 23:49:10.212032 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:10.212001    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-n8vr2_ca495489-b79a-45ad-b566-68c7090b04c9/download-server/0.log"
Oct 31 23:49:12.006273 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:12.006244    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7qrj2_13faa1a0-2c8b-4bb0-9531-10462aa1c3f4/dns/0.log"
Oct 31 23:49:12.034403 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:12.034349    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7qrj2_13faa1a0-2c8b-4bb0-9531-10462aa1c3f4/kube-rbac-proxy/0.log"
Oct 31 23:49:12.189577 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:12.189549    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cn8jx_e46ea70b-37a4-4934-920c-ecdab212ecc6/dns-node-resolver/0.log"
Oct 31 23:49:12.864781 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:12.864747    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-79bdd8d85d-qbfpp_5d845527-5948-4edd-baab-2fe6d2bcba4a/registry/0.log"
Oct 31 23:49:12.961543 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:12.961516    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xw8kl_8b1db31b-aec0-495b-8b0c-4abbde3acb28/node-ca/0.log"
Oct 31 23:49:14.925445 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:14.925331    4301 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-6228z"
Oct 31 23:49:15.854206 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:15.854168    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-689d5458bb-hzp7b_7230a094-2ba4-41af-8bec-2d8b2b266ca9/kube-auth-proxy/0.log"
Oct 31 23:49:15.920968 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:15.920936    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-6qscl_e6602036-1c07-4882-ab1a-1dbbd0a857f5/istio-proxy/0.log"
Oct 31 23:49:15.922127 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:15.922104    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-6qscl_e6602036-1c07-4882-ab1a-1dbbd0a857f5/istio-proxy/1.log"
Oct 31 23:49:16.090081 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:16.090046    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-s6nvl_99873d5d-87c2-423a-9a44-1b3fc83d0b46/istio-proxy/0.log"
Oct 31 23:49:16.206066 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:16.206036    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-z9tfc_8ea56eff-0929-4f33-9adc-f10042fb1b96/istio-proxy/1.log"
Oct 31 23:49:16.206438 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:16.206353    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-z9tfc_8ea56eff-0929-4f33-9adc-f10042fb1b96/istio-proxy/0.log"
Oct 31 23:49:18.631677 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:18.631643    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-t6v8n_2425ea7b-ae01-49db-950b-8f5268a4c0d5/serve-healthcheck-canary/0.log"
Oct 31 23:49:19.379963 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:19.379927    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mg6wn_2a85016a-a525-4512-a83f-b355b784b58e/kube-rbac-proxy/0.log"
Oct 31 23:49:19.406996 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:19.406970    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mg6wn_2a85016a-a525-4512-a83f-b355b784b58e/exporter/0.log"
Oct 31 23:49:19.436142 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:19.436112    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mg6wn_2a85016a-a525-4512-a83f-b355b784b58e/extractor/0.log"
Oct 31 23:49:19.579048 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:19.579010    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-gcst4-bgj2j_a3fb9361-cdf3-40d3-ae4b-d01fb6a569d0/insights-gathering/0.log"
Oct 31 23:49:19.749982 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:19.749892    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-689d5458bb-hzp7b_7230a094-2ba4-41af-8bec-2d8b2b266ca9/kube-auth-proxy/0.log"
Oct 31 23:49:19.785846 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:19.785817    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-6qscl_e6602036-1c07-4882-ab1a-1dbbd0a857f5/istio-proxy/1.log"
Oct 31 23:49:19.786043 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:19.786009    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-6qscl_e6602036-1c07-4882-ab1a-1dbbd0a857f5/istio-proxy/0.log"
Oct 31 23:49:19.878560 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:19.878530    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-s6nvl_99873d5d-87c2-423a-9a44-1b3fc83d0b46/istio-proxy/0.log"
Oct 31 23:49:19.933066 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:19.933027    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-z9tfc_8ea56eff-0929-4f33-9adc-f10042fb1b96/istio-proxy/1.log"
Oct 31 23:49:19.934379 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:19.934348    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-z9tfc_8ea56eff-0929-4f33-9adc-f10042fb1b96/istio-proxy/0.log"
Oct 31 23:49:22.001932 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:22.001902    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q_19ea0f64-10fa-438a-aa25-724be70d152f/manager/0.log"
Oct 31 23:49:22.028732 ip-10-0-135-163 kubenswrapper[4301]: I1031 23:49:22.028710    4301 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7c5c6d88b7-b8c8q_19ea0f64-10fa-438a-aa25-724be70d152f/copy-manifests/0.log"